Simply using a given branch doesn’t really do it. I’m seeing stuff thats too young and too old for my project.
Do I just import derivations? Do I do a nix-env -iA xxxxxx? If so, what should be working. I’ve yet to be able to nix-env -iA clang_7_1_0 (for example) and get anything useful. I’ve tried a number of variants. The funny thing is how few examples of that there are in the documentation.
Most packages I’ve seen care a lot about the versions dependent packages, and often have very specific needs.
You can see the available versions using the package search. Not all packages will have every version, some don’t have more than a single version available.
Another option is using a specific nixpkgs revision, either for a single package using an overlay or for your entire config or derivation. This only works if the version you want has been packaged before.
Lastly you could override the version on a package, but this would likely mean the package gets built from source instead of being downloaded from the binary cache.
it’s not really desirable to have all versions specified. We have a lot of infrastructure building, and caching these packages. Have a N x M grid of packages for every version increases complexity and resource usage.
In your clang example, why is clang_7 insufficient?
In production work (not even high security) it is very common for specific versions of each package in the list of dependencies because of specific fixes, changes in api etc.
For example, dcos builds packages by exact versions. I’ve seen when there were problems with their build system, the result was numerous difficult to isolate issues.
In high security work, doubly so because the dependencies need to have their source audited, compilers must have needed mitigations, etc.
This is a real requirement. I have some code for ransacking the channels to get the needed derivation paths, and that code is actually pretty sweet, but it doesn’t seem like this is offbeat. How many projects don’t need a specific compiler version that has a new mitigation?
Referring to an exact version, and querying the version of a package are two different things.
Querying build dependencies:
nix-store -q --tree $(nix-instantiate -A <package>)
Querying runtime dependencies:
nix-store -q --tree $(nix-build -A <package>)
if you just wanted a flat list, then you could remove the tree formatting, the nix/store and hash, then do cat list | sort | uniq
# remove tree formating, then remove /nix/store/hash, then remove [...] from the end. Then sort and make unique
$ nix-store -q --tree $(nix-build -A python3) | sed -e 's/^[-+|\ /]*//g' | sed -e 's/^[^-]*-//g' | sed -e 's/\ \[\.\.\.\]$//g' | sort | uniq
bash-4.4-p23
bzip2-1.0.6.0.1
expat-2.2.10
gdbm-1.18.1
glibc-2.32
libffi-3.3
libidn2-2.3.0
libunistring-0.9.10
ncurses-6.2
openssl-1.1.1h
python3-3.8.6
readline-6.3p08
sqlite-3.33.0
xz-5.2.5
zlib-1.2.11
there’s nothing stopping you from creating an overlay with versions of packages that you explicitly want. incase you do want to add a clang_7_0_1.
It just doesn’t make sense for nixpkgs to have such a large amount of “intermediate” versions. Generally we are more concerned about breaking changes, thus only carry major (or major+minor, e.g. python37) versions for toolchains.
In all cases, the problem is not querying as such.
Nor is it acceptable to take what nix decided I need.
I need to define the version of the package, not know what I was given.
For any current project I’m aware of, deciding I need clang-7.1.0, openssl-1.1.1g, glibc-2.27 not whatever version is present in nixpkgs branch nixos-19.09 or whatever is a requirement.
For any package I’m aware of, there is a suitable derivation in at least one of the channels, but again, I need to be able to specify those versions and have them remain constant regardless of the surrounding environment.
This is a requirement for building any production code, period, so far as I am aware, so if it cant be done, thats an issue.
I think currently it can be done at great effort and I suspect that is bad if folks want this system to be adopted. It should be simple.
let clangOverlay = self: super:
{
clang_7 = super.clang_7.overrideAttrs (oldAttrs: rec{
version = "7.1.0";
name = "${oldAttrs.pname}-${version}";
src = ... ;
};
};
in import <nixpkgs> { overlays = [ clangOverlay ]; };
clang is a bit more involved due to llvmPackages, so you will most likely have to pin those as well.
It’s do-able to pin exact version, but not without some effort on your part.
19.09 has been EOL for 6+ months, why are you using that?
For production software, you should be using the latest release, which is 20.09 in this case.
[11:22:12] jon@nixos ~/projects/nixpkgs (release-20.09)
$ nix-build -A clang_7 -A openssl -A glibc
/nix/store/lfgabqsrb0ja2snr8aalibway4jp5mmk-clang-wrapper-7.1.0
/nix/store/q0ajpzppqfrlbzbddbbzv1w6vfzydhk5-openssl-1.1.1g-bin
/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31
nixpkgs is a single coherent package set, it avoids multiple versions as much as possible. First, because it is an additional maintenance burden, second because carrying multiple versions can lead to problems (e.g. multiple incompatible versions of Python modules ending up the Python module path). If you really need specific versions of packages, it’s best to follow @jonringer’s advise to make your own package set. You can copy the derivations from nixpkgs and alter them as needed.
If you are using custom combinations of a C compiler, glibc, etc. it’s unlikely that Hydra built that custom constellations of packages, so there is not really a way around it. Packages in nixpkgs are not built in isolation. If you e.g. replace gcc by another version, every package that has gcc as a transitive dependency needs to be rebuilt.
For any package I’m aware of, there is a suitable derivation in at least one of the channels, but again, I need to be able to specify those versions and have them remain constant regardless of the surrounding environment.
No, there often is no suitable derivation in any channel, because the version from the channel is strictly tied to the then-in-the-Nixpkgs versions of transitive dependencies. If you have constraints on all version, openssl linked against wrong-for-you glibc that cannot be just installed together with desired glibc version is not good enough for your use case.
So you probably do need to build all of your dependencies (on the other hand, if changing dependency versions is a non-trivial amount of vetting, maybe building and caching them locally is not that bad…)
This is a requirement for building any production code, period, so far as I am aware, so if it cant be done, thats an issue.
This is an overgeneralisation if claim universality… (although sure, you set of requirements is not unique, just not the primary focus of Nixpkgs)
I tried following the video etc.
Not quite clear what the problem is here.
let
packageOverrides = self: super: {
/* We need glibc to be 2.27 so we can run on ubu 18.04. glibc must always be a lower revision than
the platforms glibc or it will get an undefined symbol */
glibc = super.glibc.overrideAttrs (oldAttrs: rec {
version = "2.27";
name = "${oldAttrs.pname}-${version}";
src = builtins.fetchurl {
url = "https://ftp.gnu.org/glibc/glibc-${version}.tar.xz";
sha256 = "13c8dlmv1vgfx53fr7m7y8d4rcw4alb9xjznns7z6jv9l8s0ak3z";
};
});
};
in
self: super: {
glibc = super.glibc.override { inherit packageOverrides; };
}
in “~/.config/nixpkgs/overlays/glibc-overlay.nix”
produces an error when built:
brcamp@ubuntu:~/nixpkgs/pkgs/development/libraries/glibc$ nix-build -I/home/brcamp -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'
error: anonymous function at /home/brcamp/nixpkgs/pkgs/development/libraries/glibc/default.nix:1:1 called with unexpected argument 'packageOverrides', at /home/brcamp/nixpkgs/lib/customisation.nix:69:16
(use '--show-trace' to show detailed location information)
self: super: {
/* We need glibc to be 2.27 so we can run on ubu 18.04. glibc must always be a lower revision than
the platforms glibc or it will get an undefined symbol */
glibc = super.glibc.overrideAttrs (oldAttrs: rec {
version = "2.27";
name = "glibc-${version}";
src = builtins.fetchurl {
url = "https://ftp.gnu.org/gnu/glibc/glibc-${version}.tar.xz";
sha256 = "0wpwq7gsm7sd6ysidv0z575ckqdg13cr2njyfgrbgh4f65adwwji";
};
});
}
Got me to building, but the build doesnt work.
Hunk #1 succeeded at 1625 (offset 112 lines).
applying patch /nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch
patching file sysdeps/unix/confstr.h
applying patch /nix/store/xkd50xxii6k7l1kmw4l5x6xzbhamcs87-allow-kernel-2.6.32.patch
patching file sysdeps/unix/sysv/linux/configure
patching file sysdeps/unix/sysv/linux/configure.ac
patching file sysdeps/unix/sysv/linux/dl-osinfo.h
applying patch /nix/store/3l70d7kcfsh91w6792h4fqs4kjbq17py-glibc-reinstate-prlimit64-fallback.patch?id=eab07e78b691ae7866267fc04d31c7c3ad6b0eeb
patching file sysdeps/unix/sysv/linux/getrlimit64.c
patching file sysdeps/unix/sysv/linux/setrlimit.c
Hunk #2 succeeded at 46 with fuzz 1.
patching file sysdeps/unix/sysv/linux/setrlimit64.c
Hunk #1 succeeded at 36 with fuzz 2.
applying patch /nix/store/nqa9d4yxz1l2cgswzqr1pkm4jfrksm0q-locale-C.diff
patching file localedata/locales/C
applying patch /nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch
patching file elf/dl-tls.c
Hunk #1 succeeded at 818 (offset -9 lines).
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/inet/idna.c
|+++ b/inet/idna.c
--------------------------
File to patch:
Skip this patch? [y]
Skipping patch.
1 out of 1 hunk ignored
builder for '/nix/store/i40hdhlvnig75qw5c5g3phc75ds90xf7-glibc-2.27.drv' failed with exit code 1
cannot build derivation '/nix/store/0sna51pm54nxb5panigi84kdh2rhi32s-glibc-2.32.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0sna51pm54nxb5panigi84kdh2rhi32s-glibc-2.32.drv' failed
I got it to build by telling it not to patch but that doesn’t seem right.
The thing is, there is a derivation around for 2.27 from 19.09. But I can’t select it because there is only the 20.09 2.32 that prevents me from running on anything other than nixos. This seems like a lot of work to go through to get what seems like a pretty generic expected outcome, ie that a package built with nix should be able to be run on whatever current linux.
Re: patches — I guess you can copy the patch list from the old expression?
Re: 2.32: what’s the specific problem with running Nix-built packages using some glibc on a system where the default is a different glibc? Nix packages won’t use system glibc either way. True, LD_PRELOAD of system libraries into Nix binaries or vice versa won’t work.
Glibc links an external symbol (for example GLIBC_232) which is checked on load. The symbol is defined for each version of glibc starting with the first, extending to the latest. You will get a good link if the client glibc version is less than or equal to the platforms glibc, but running it will fail if the version being used is newer.
So, link in glibc-2.32 to your binary, running it on fedora 33 or ubu 20.04 will get “GLIBC_232 symbol not found”.
This is deliberate and by design. It is not recommended to try to upgrade the glibc on the platform, even if it was something folks would be willing to try.
The purpose in this case of building in nix is to provide a reproducible and auditable build of a .deb or .rpm package to be run on the usual systems. Nixos and the nix store do not enter into the ultimate product.
