Hey!
OpenSSL 1.1.1u was released two weeks ago to fix some vulnerabilities. It was backported to 22.11 with [Backport staging-22.11] openssl_1_1: 1.1.1t -> 1.1.1u by github-actions[bot] · Pull Request #235017 · NixOS/nixpkgs · GitHub but since then, it is still in staging. This is just an example, but vulnerabilities take time to reach users, even without taking a detour to staging.
I don’t really have a solution but I was wondering if there is some work in-progress to deal with this?
I’m not aware. One full rebuild iteration takes normally roughly a week, and each of them fixes CVEs. We’re now supporting three separate branches for about a month or two, so a lag of three weeks is normal to happen, I’m afraid. Right now 22.11 has the lowest priority of the three AFAIK.
BTW, there’s replaceRuntimeDependencies as well, e.g. see OpenSSL 3.0.7 update (2022-11-01) FAQ
Thanks! The tool mentioned to track how far is the PR in a release is quite useful!
There’s also a labeltracker as an RSS feed: https://labeltracker.eno.space/nixpkgs-security/prs.xml and Issues labeled `1.severity: security' in NixOS/nixpkgs which you can subscribe too.
I am working with the author and ideally the new infra team or nix-community, so we can promote it a bit better.
But work in this area definitely requires help.
So, for this particular case, let’s do 22.11 rebuild now. You can follow:
https://github.com/NixOS/nixpkgs/pull/238445
For those interested, this is what it currently shows for the backport PR linked in the OP: https://nixpk.gs/pr-tracker.html?pr=235017
Sounds like it is about time to revive Shipping Security Updates by nbp · Pull Request #10851 · NixOS/nixpkgs · GitHub …
When I created this branch, people told me that this was not worth the complexity because security updates were shipping the next day.
I expect it’s still the same for the bulk of people fixing security issues in nixpkgs, but certainly it’s good to link that work.
Even if security updates were shipping the next day via Hydra, not everyone has the same resources to build that many packages. Your work would additionally make it viable to apply for example bugfix patches to often used library packages that aren’t likely to land in either upstream or nixpkgs any time soon, without having to wait days for every single update to build. It’s sad to hear that that is the reason why you’ve stopped working on it and I hope you reconsider.
In trying to follow your pr as best as I could I also found this explanation on the www:
https://www.theshortlog.com/posts/nix-security-updates-update/
I could understand that one a bit better 
Seems to me that in the years since then the need for speed is obvious, how do we manage anno 2023?
An (albeit a bit radical) approach might be to fix the dependency graph in order to better represent ABI-compatible updates:
Right now, a huge and growing number of packages are reverse dependencies of both interface and implementation of security critical libraries like libc or openssl. But this is only necessary as long as we we consider hardcoding runtime linkage as an important part of the immutability concept.
Instead at least security critical base libraries could be packaged in a way where only the header files defining the ABI, and possibly some stub library to make it easy for linkers, fill a root position in the dependency graph as an interface package. Both the consumers of the library functions, and the library implementation itself, would be reverse dependencies of the interface package.
In order to have usable packages, either the runtime environment would have to configure the runtime linking against the implementation package, or there would have to be wrapper packages that binary patch the implementation package path into the already built package that links against the stub library.
@gm6k I’ve thought a bit about this approach and have a few questions that I couldn’t figure out how to resolve, so I’m interested in hearing your thoughts.
Instead at least security critical base libraries could be packaged in a way where only the header files defining the ABI, and possibly some stub library to make it easy for linkers, fill a root position in the dependency graph as an interface package. Both the consumers of the library functions, and the library implementation itself, would be reverse dependencies of the interface package.
My mental model of this works best with content-addressed store. Say you have foo-glibc-1.23.4 and bar-glibc-1.23.4 which differ in only a security patch. It would be possible to extract baz-glibc-abi-1.23.4 which is the same for both foo and bar consisting of both the interface headers and a stub ELF binary dynamically generated from the exports of glibc’s binaries, and then link against it.
In order to have usable packages, either the runtime environment would have to configure the runtime linking against the implementation package, or there would have to be wrapper packages that binary patch the implementation package path into the already built package that links against the stub library.’
This is where I got hung up. I think it is impossible to securely pick a proper implementation package even with content-addressed store without solving the Trusting Trust problem.
Consider what happens if a package mypkg links against baz-glibc-1.23.4. We could use some path like /nix/lib/baz-glibc-abi-1.23.4 that is a symlink into the store to the “latest” version of glibc-1.23.4. (note how cursed this is - nix is basically performing a hash-based update-alternatives Debian style). But which glibc-1.23.4 implementation do we choose, is the question. A possible attack is that one could locally build an implementation of baz-glibc-abi-1.23.4 that someone locally who is not a trusted user built, and that could contain a backdoor. So you would need to somehow trust that the only links into /nix/lib are allowed to be there, somehow, and I’m not sure where to start with that.
Thanks for the feedback! It makes me think I communicated my suggestion too unclear. I don’t mean to abandon hardcoded runtime linkage in the sense that the package has to dynamically find the latest library for running. But we would need to abandon the idea that the package that consumes library functions has to know at build time to which library version it will be linked later on.
At least at run time, the whole environment should still be immutable and secured by hashed derivations.
Consider the following simple scenario with one library lib and one package pkg1 using it. When the lib derivation changes, both lib and pkg1 have to be recompiled:
Now (sketching the second variant using a wrapper package), we put the library interface (meant to change seldomly) into a separate package lib_interface. The previous pkg1 build derivation becomes pkg1_build. It is not meant to be ran directly because it is linked against the interface package at build time. pkg1 becomes a wrapper derivation which can be realized quickly because all it does is taking pkg1_build and replacing all references to lib_interface by references to lib. It is marked yellow to indicate it won’t contribute to compilation burden significantly:
When lib has to be updated in an ABI-conforming manner, only the greyed derivations have to be rebuilt because lib_interface is unchanged. Since pkg1 can be built quickly, only lib is significant to the compilation burden:
The same should work if the runtime environment is responsible for setting up the linking. Still, the (authoritative) package database defines which packages to use. Packages built by untrusted users are expected to end up with a different hash, and don’t get unintentionally picked up.
sorry for the noise if this is naive/stupid/evil:
Why can’t the contents of the vulnerable store path be “replaced” with the contents of the fixed store path using something like a bind mount until everything can be properly rebuilt?
(create a new view of the store where the vulnerable store path points to the inode of the fixed store path)
bwrap --dev-bind / / --ro-bind /nix/store/...fixed /nix/store/...vulnerable command
Guix provides graft for security updates, so you don’t have to recompile everything except in case of ABI change (which shouldn’t happen on security fixes) Grafts, continued — 2020 — Blog — GNU Guix
replaceRuntimeDependencies is better than that. (It keeps purity while avoiding rebuilds.) Some people do call it grafting, too.
Is something along these lines what you imagine:
/nix/store/blah-app depends on libfoo. libfoo has a CA output /nix/store/aaaa-libfoo-abi where it installs, for example, header files.
Then app’s rpath, it wouldn’t link against /nix/store/bbbb-libfoo/lib/libfoo.so directly but /nix/abi/aaaa-libfoo/lib/libfoo.so which in turn is pointed at some version of libfoo whose abi output is also /nix/store/aaaa-libfoo-abi.
I uploaded a basic PoC which illustrates the approach for patching openssl upgrades into direct reverse dependencies: (implementation readme)
I’ve been mulling over this interface/implementation separation notion, and it’s given me an odd idea.
Linux distributions other than NixOS have already solved the problem of separating out the interface of a library from its implementation: the ABI name of a library, as part of its path in /usr/lib, serves as the interface, and the contents of that library the implementation. Nix abandons this structure so that it can have all the nice things that drew us to Nix. Nix packages depend directly on their dependencies’ implementations, which makes rebuilds costly.
But many packages in Nixpkgs are two-part (cheap) wrapper and (expensive) wrapped pairs. One advantage of that sort of split is that dependencies that can be isolated to the wrapper layer don’t cause the wrapped package to rebuild.
What if we generally used this pattern to provide library dependencies? What if Nix compiled binary packages that reference /usr/lib/libwhatever.so.12 like every other distro does, but all the user-facing executables came from wrapper packages that:
/usr/lib/libwhatever.so.12 /nix/store/...-libwhatever-3.4.5/lib/libwhatever.so.12
The final piece of the puzzle would be allowing library mappings to be provided at build time impurely, without appearing as part of the derivation (so that compilers have some concrete implementation to build against). Instead, derivations would contain only the interface half of the mappings, because if that half changes then the package needs to be rebuilt. (I know, getting an impure feature into Nix is probably an insurmountable mountain, even relative to making global changes in Nixpkgs.)
The result: when a library needs to be patched, it gets rebuilt. Packages that depend on that library only get rebuilt when the library’s ABI name changes; otherwise, it’s only the mapping files in the wrapper packages that point directly to the library’s store path, and rebuilding wrappers is cheap. So security patches get shipped much faster.
But we still keep everything we like about Nix: multiple versions of a package can coexist, system rollbacks are atomic, there isn’t some special location in the filesystem we have to trust and protect outside of the Nix store.
Extra advantages:
Interesting! Patched loader seems most realistic to me, but I’m not in very deep.
What do current wrapped packages like firefox use?
This sounds awesome, but I’m not entirely sure how it would work? Can you change the loader path without patching the binary?