Designing relocatable shebangs

Development
1

Linux 7.3 brings an incredibly powerful feature: eBPF binfmt_misc hooks. You can see some discussion on this here, especially the blog post by @fzakaria here.

One idea is to make the PT_INTERP elf header support $ORIGIN, so that binaries are completely relocatable. However, this breaks compatibility with older Linux kernels, and with users that otherwise cannot enable BPF.

A solution is to define the normal PT_INTERP as /nix/store/..., but to additionally define a separate PT_INTERP_NIX section that replaces the absolute path with a relative path. On a Linux 7.3+ kernel with the bpf program loaded, the binary will be relocatable, but on older systems everything would still work as long as the binary is installed into the global path. I think this is a good path forward for Nix.

But for full compatibility, the same thing needs to happen for shebangs.
The binfmt_misc program can also recognize the execution of scripts that start with #!, and can rewrite them as necessary. One option is to just substitute $ORIGIN in the shebang, but this again would lose non-bpf compatibility.

I propose the following protocol: binaries that start with #!\t (that is 0x23, 0x21, 0x09, 0x20) will be marked as “relocatable”. Ie. the bpf match program will return true when the file starts with those 4 characters.

When the bpf program recognizes a script, the interpretation rules are: skip all characters from the start of input until the first newline. Then skip the following sequence of non-whitespace characters, then the next sequence of whitespace characters. After that, the rest of that line will be parsed as a shebang, but will have $ORIGIN replaced.

For example:

#!	 /nix/store/xxxxxxxxxx-python3-3.14.8/bin/python3.14
# $ORIGIN/../../xxxxxxxxxx-python3-3.14.8/bin/python3.14
print("Hello from Python!")

#!	 /nix/store/xxxxxxxxxx-lua5_5-5.5.1/bin/lua
-- $ORIGIN/../../xxxxxxxxxx-lua5_5-5.5.1/bin/lua
print "Hello from Lua!"

#!	 /nix/store/xxxxxxxxxx-bash-5.3p20/bin/bash -e
# $ORIGIN/../../xxxxxxxxxx-bash-5.3p20/bin/bash -e
echo "Hello from Bash!"

I think that using a recognizable yet uncommon sequence of whitespace (\t ) is a good way to implement this feature, but let me know if you have any thoughts!

7 Likes
2

That makes sense to me; we should probably upstream the new eBPF PT_INTERP_NIX BPF Handler as a NixOS module now that 7.3 is out;
I had been meaning to do that…
If you want to take a stab at that, you can add me as a reviewer?
(We can then propose the same to Debian and Fedora?)

The kernel should include an example for Nix BPF in the source AFAIR (it was part of the patch series we worked on)

Here is the eBPF for the shebang: shebang_origin.bpf.c · GitHub

2 Likes
3

Do we need to specially mark them? Can’t we just have the bpf program look for #!/nix/store/ and substitute in #!$ORIGIN? I can’t think of a case where this would break scripts that haven’t opted in, but that could of course be a failure of imagination on my part.

3 Likes
4

I don’t have better idea, but I imagine marking with tab (\t) could break (in a non-obvious way) when converting between spaces and tabs in editor. Also inputting \t would be slightly annoying if editor uses spaces as tabs.

Edit: I just realized that these sequences would be written by patchShebangs, and the file kept read-only in nix store, so my concerns are probably not an issue.

5

I think it could be a speed improvement, as a Linux 7.3 system will still not have that many fully-relocatable scripts on it while it will definitely have a lot of scripts with a #!/nix/store shebang.

I could also imagine someone wanting to quickly edit a script in the store, so they want to cp /nix/store/xxxx/script.py ./script.py, edit, then run. By only performing the replacement for the #!\t sequence, you can prevent the bpf replacement from running by just deleting one of those whitespace characters.

1 Like
6

This is a good point. And while NixOS will only be writing these files using patchShebangs, I think the concept is much more general and could be hand-written for AppImage-style uses.

Maybe we could also recognize sequences like “17 space characters” or “7 tab characters”, where no human would write it themselves but would survive code editors.

1 Like
7

I think we should try to name it PT_INTERP_ORIGIN or PT_INTERP_RELOC, since it’s useful beyond Nix (I know the GNOME OS folks were at least open to using it with Toolpak, and IIRC bazel was also interested).

And I can try to take a stab at it next week.

3 Likes
8

Wouldn’t the eBPF hook check that the script itself is in /nix/store/ before doing a replacement? So copying it would automatically disable it.

EDIT: of course thinking about it for another second, that breaks the relocatability which is the whole point… But you could still match on /<hash>-.../ which is probably enough

9

I think its a bad idea to strongly couple the bpf program with Nix, let alone /nix/store. I think it’s much less of a challenge to get the feature added to Ubuntu/Fedora if Bazel/Buck2/Toolpak are all interested in the same feature.

3 Likes
10

the BPF program can be NixOS-specific I think; if there is something to leverage a particular distribution to do in the BPF that is smart.

I agree though that there can also be a “general” approach; it’s a handler chain so there can be the “default BPF $ORIGIN” handling and if one is needed, a NixOS optimized one?

Feel free to reach out to me if you want to collaborate.
I am expecting baby #4 (this week!) but I still like to hack :stuck_out_tongue:

2 Likes
11

We ought to try to get whatever we pick into FHS 4. @zwol any thoughts? :grinning_face:

See “store-based” here:

Distributions conforming to the “store-based” profile use a unique installation prefix for each software package that they distribute. An installed system that uses a store-based distribution will rely on search paths and symlink farms to present a unified view of available software.

Maybe this could use something about $ORIGIN? It seems to prefer absolute paths…

In a store-based distribution, each package’s installation prefix is a subdirectory of the “store root,” an absolute path defined by the specific store-based distribution.

In any case, it will be much easier for FHS distros to support if this is just canonized in the FHS.

1 Like
12

FHS 4 seems to have stalled out. I would still like to see it happen but, well, I posted that draft and got one piece of feedback, from the guy who originally wanted to do FHS 4, which boiled down to “you and I have very different visions for what the standard should even be”, and since then nothing, for the past eight months.

The vagueness about what path the “store root” actually is, is intentional, because there’s at least two candidates in use “in the wild” (/nix/store for Nix and /gnu/store for Guix), and it isn’t, IMNSHO, FHS’s place to pick a winner. Genericizing it (/store perhaps) isn’t a solution; that would break both Nix and Guix, and even if it didn’t, would rule out future use of either as an overlay upon the other.

Side note, I firmly believe that eBPF is a security disaster waiting to happen and I always turn it off at kernel build time. I would turn off original BPF as well if kconfig would let me. I do not want to see any technology based on eBPF become a hard requirement for NixOS.

3 Likes
13

FWIW, my immediate thought is Android, which requires hideous hacks to support /nix/store with proot for nix-on-droid, and would also never enable BPF to do this kind of thing in a million years, either. (So they are forcing new, worse hacks…)

I do wonder if they’d be amenible to a Kconfig option to enable it, then vendors could opt in. We’d have to convince Google for Android but I feel like that’d be easier than convincing them to turn on BPF.

edit: I was mistaken, they do actually enable it: https://source.android.com/docs/core/architecture/kernel/bpf - that’s how vendors do things like traffic classification, actually

1 Like
14

I don’t see how this is a meaningful speed improvement, since the ebpf hook will always need to run to check for #/\t so searching for or replacing a longer string is a difference of a few microseconds.

Why would you want to disable the replacement when running an external script? surely you still want your copied script to replace /nix/store/… with wherever your local store actually is so it actually still works.

I guess besides the ick factor of the hack that is using #!\t as the magic bytes for PT_INTERP shenanigans (!#?!), I’m really failing to see why this is needed. That’s not to say that this proposal is unnecessary or misguided, just that I’m failing to grok why this design is the one to pursue.

Alternatively, one could use the same trick that nix-shell uses to wrap scripts, providing a chain-loading program makes it 100% clear to to the reader that the interpreter is being redirected

#! /usr/bin/env nix-chain-load
#! $ORIGIN/../../xxxxxxxxxx-python3-3.14.8/bin/python3.14
5 Likes