Simplest way to solve binaries relocatable problem, instead of $ORIGIN

Two simple rules:

  1. Ldso’s searching order is only exe‘s directory (not current directory) and PATH, remove everything else such as absolute path, RPATH, RUNPATH, LD_LIBRARY_PATH, LD_PRELOAD, ld cache …
  2. Add kernel parameter “ld=” to customize ldso, override PT_INTERP.

Now I can easily have multiple core systems at same time, because everything is portable. For example, I put all binaries into /linux_1, and set boot parameters to “init=/linux_1/init ld=/linux_1/lib.so”. I can create another /linux_2 /linux_3… as I wish, just like Windows, no built-in absolute path any more. Core system can even be made ‘immutable.’ Everything else including GUI will be placed in user’s directory as user software. PATH will be set by user as they like. Simple means robust, no ‘backdoors,’ also means user-friendly, just as simple as DOS.

See Linux Purification Plan - Make Linux portable for details.

Subtract. Bring Linux Back to KISS.

You understand the point of PT_INTERP is to support multiple linkers, right?

1 Like

Of course, if you have complex needs, you can leave this parameter empty.

Now submit it as a patch upstream and also get it accepted :wink:

1 Like

Does it mean you can’t really test binaries against a different ld.so on such a system at all? No chance to build and test newer libc? No chance to create a chrootwith a different ld.so?

I think there is a more fundamental architectural issue here, beyond simply choosing a different dynamic linker.

Look at the Windows NT architecture. ntoskrnl and ntdll.dll form a relatively stable system foundation. The kernel is responsible for core process and image management, while NTDLL provides essential user-space system functionality, including the user-mode loader. Higher-level APIs such as Win32 and C runtime libraries are built on top of this foundation. They can evolve independently without fundamentally changing the executable loading architecture.

I think this separation of responsibilities is much cleaner.

There are two things about the Linux ELF architecture that I find particularly problematic:

1. Why are static and dynamic executables loaded through fundamentally different paths?

The kernel directly loads a static executable, but for a dynamically linked executable, it hands control over to a user-space ld.so, which then takes responsibility for loading the remaining dependencies and preparing the program.

In my opinion, executable loading should be a unified system mechanism. Whether an executable is static or dynamic, it should go through the same fundamental loader. The difference should simply be whether additional shared libraries need to be loaded.

2. Why is the dynamic linker so closely tied to libc?

A dynamic linker is a fundamental system component. Its responsibilities include loading shared objects, symbol resolution, and relocation. These are core executable-loading functions, not ordinary C library functionality.

Libc, on the other hand, is an application runtime library. It is perfectly reasonable for libc implementations to evolve, change, and be replaced. But the fundamental executable-loading mechanism should not be tied to the development cycle or implementation details of a particular libc.

Ideally, ld.so should be an independent, stable system component, with a small and well-defined interface to the kernel. It should not depend on a particular libc implementation.

This is why I find the Windows NT separation between the kernel, NTDLL, and higher-level runtime libraries interesting. I am not suggesting that Linux should copy Windows internally, but I do think the underlying separation of responsibilities is worth learning from.

Regarding your question about testing different linkers: I don’t think this requires making every executable independently choose its own interpreter through PT_INTERP. Explicitly invoking another linker is a much more straightforward way to test an alternative runtime environment:

/path/to/ld.so ./program

Of course, this is not a complete substitute for testing the normal kernel execution path. But it demonstrates that alternative linkers can still be tested without making per-executable linker selection a fundamental system requirement.

I have always felt that Linux has accumulated too many implicit behaviors and overlapping mechanisms. Consider chroot, for example. It is useful for creating an alternative filesystem environment, but it is not a security boundary. Its behavior is easy to misunderstand, and it can create a false sense of isolation. More mechanisms do not necessarily mean better security; they can also make systems much harder to understand and audit.

My goal has always been to simplify Linux: use fewer mechanisms to accomplish the same things, preferably in a more predictable and reliable way.

To be clear, I am not claiming to have a complete implementation of this architecture. My current ld= kernel parameter and modifications to musl’s ld.so are only pragmatic experiments toward this direction.

What I really want is a cleaner separation of responsibilities: the kernel provides the execution foundation, ld.so provides a stable and independent executable-loading mechanism, and libc remains an ordinary, replaceable user-space runtime library.

Do subtraction, get Linux back to KISS.

1 Like

Sorry, it’s too much text for me.

If I read you correctly you want to add extra code into kernel to override pure userspace INTERP policy of what interpreter to use. AFAIU you do recognize it will immediately break current use case of multiple interpreters. muslbinaries would break outright on glibc system. 32-bit binaries would break on 64-bit system. Does it sound about right?

It’s not really clear what problem you are trying to solve.

You mention relocatable binaries, but interpreter sounds like the easiest problem you have. And your solution make things worse for existing systems (like 32+64 bit binaries mix). Userspace is full of absolute paths. For example ld.soyou mention has paths like /usr/lib, which imply ABI (and implementaion detail) of libc. In non-ELF world there is /usr/sharelike paths that you have to modify in the binary if you move binaries around.

2 Likes

I have occasionally thought along the same lines. There is, unfortunately, a pretty nasty catch: The dynamic linker and the lowest levels of the thread library are tightly coupled, because loading libraries is a process-wide operation that can be triggered at any time (perhaps nigh-simultaneously from multiple threads) and that calls back into user code (via e.g. C++ static constructors), and because some high-level language features require support from both the threads library and the dynamic linker (e.g. thread-local variables).

The dynamic linker, seen as an independent program, is also a large, complicated Twinkie that uses a great deal of the C library as any application might, including relatively high-level things like malloc and stdio.h.

This (plus Windows’ own multi-decade history of accreting features) is probably why, if you run the equivalent of size -A on ntdll.dll, you will discover that it contains a larger amount of machine code than GNU libc’s libc.so, ld.so, and libpthread.so all put together!


This isn’t to say that you shouldn’t keep pursuing your idea, but I think you should pursue it as an independent “upstream” project, not as part of NixOS. And I suggest that you should be looking for ways to make your language-independent dynamic loader run as a separate process, an independent system-wide daemon, from the programs it loads. This will require creation of new kernel interfaces (it’s currently not possible to do all the things to a separate process that ld.so does to its own process), and it involves an awkward chicken-and-egg problem at boot time, but I think it’s the only way to break the coupling.


p.s. If you are serious about “do subtraction, get linux back to KISS” you need to be thinking about ways to do less in the kernel. Making the kernel’s binary loaders do more things is contrary to this goal.

3 Likes

I think we’re approaching this from different design goals.

My goal is a simpler, self-contained system:

  1. The system core, including its configuration, should live in a single relocatable directory. Multiple versions could coexist, just like Windows installations.

  2. Each user application, including its own configuration and dependencies, should also live in a single relocatable directory, like portable Windows applications.

  3. All executables, static or dynamic, should go through one unified loader. This could be integrated into the kernel’s binfmt_elf, or implemented as a separate, stable system component, similar to the relationship between ntoskrnl and ntdll.dll.

Different architectures and libc implementations are simply things the loader should handle, not reasons to abandon a unified loading model.

Ultimately, I want to eliminate hardcoded absolute paths and unnecessary filesystem conventions.

The ld= parameter is just my current experimental approach, not the final architecture.

I think we’re approaching this from different architectural philosophies.

One problem I see in the free software ecosystem is the tendency to maximize user choice at every layer. But there’s a hidden cost: the complexity of those choices is pushed upward, onto the developers and users of higher-level components.

Allowing multiple independent implementations at every layer may seem to maximize freedom, but it also creates a combinatorial explosion of configurations and compatibility issues. Eventually, developing and using software becomes increasingly difficult.

I prefer a different principle: strict, well-defined interfaces between layers, with freedom of implementation within each layer.

A system doesn’t necessarily need multiple libc implementations. Choosing one as part of the base system is perfectly reasonable. Applications could still carry their own dependencies when necessary. What matters is that the boundary between the loader and the runtime is clearly defined.

Regarding threads, I agree that the dynamic loader and the thread runtime need close cooperation, especially for TLS, concurrent library loading, and library initialization. But cooperation doesn’t necessarily mean tight implementation coupling. I would prefer a well-defined interface between the loader and the thread runtime, rather than making the loader an extension of a particular libc.

A system can choose one libc and thread runtime as part of its base environment. There’s no need to support every possible combination. What matters is that the loader depends on a clearly defined interface, not the internal implementation of a particular libc.

This is also why I want the loader to be an independent system component. It should have a minimal runtime of its own rather than relying on libc being initialized.

My goal isn’t to maximize the number of choices available at every layer. It’s to build a coherent system where complexity is contained within the layer responsible for it, rather than passed on to everyone above it.

You want this in nixpkgs?! Would you keep the nix store?

I think there’s a misunderstanding here. There are two separate things being discussed:

  1. My original proposal was simply to offer a minimal alternative to $ORIGIN expansion in PT_INTERP. A kernel ld= parameter, combined with a small modification to musl’s ld.so, could achieve this. Distributions could implement it themselves without waiting for an upstream kernel patch. That’s all I was proposing initially.

  2. The subsequent discussion was about my broader ideas on simplifying and “purifying” Linux, along with experiments on my own system. These are architectural ideas, not proposals for how NixOS or nixpkgs should be designed.

Neither is an attempt to replace the Nix store or change nixpkgs’ packaging model.

3 Likes

Thanks. That answers my question.

So you like the general movement towards flatpaks/snaps/appimuges?

1 Like

Yes, the goal is similar: making applications self-contained. My idea is closer to portable Windows applications — an application and its dependencies living together in a relocatable directory, without requiring a separate packaging framework.

More broadly, my philosophy is subtraction rather than addition: instead of adding more layers to work around the system’s complexity, I’d rather simplify the underlying architecture so that this kind of portability becomes a natural property of the system.

1 Like