Thankful for autoPatchelfHook for library dependency tweaks

Development
1

New to nix, I have been wrestling with creating some derivations I need for work, such as one for Microsoft’s sqltoolsservice.

I am so grateful to the folks who have worked on autoPatchelfHook. It is remarkably easy to use. I did the following:

  1. added the packages providing the needed libraries to buildInputs
  2. provided a list of libraries to skip in autoPatchelfIgnoreMissingDeps because autoPatchelfHook will stop at the first failure otherwise
  3. In postFixup, I added some patchelf --add-needed and patchelf --replace-needed commands. With postFixup running before autoPatchelfHook, this allowed fixing some associations so that autoPatchelfHook will then link the correct libraries. Yay for patchelf.
  4. Made sure nativeBuildInputs had autoPatchelfHook

Everything else was automatic.

This involved days of figuring out various dependency problems. Essential for this was regular searching of the entire nixpkgs repo. Especially helpful were build-dotnet and powershell and aws-workspaces. These .NET packages had examples of critical workarounds for various issues I encountered.

I guess I just wanted to tell someone about the (eventual) success. And offer these tips for other new people who are wrestling with .NET issues.

Here is the derivation, if interested:

{ lib, stdenv, fetchurl, autoPatchelfHook, pkgs, ... }:
stdenv.mkDerivation rec {
  pname = "ms-sqltoolsservice";
  version = "4.11.0.10";
  dotnet_version = "7.0";

  dontUnpack = true;

  buildInputs = with pkgs; [
    stdenv.cc.cc
    zlib
    curl
    openssl
    lttng-ust
    icu
    libkrb5
  ];

  dontStrip = true;

  nativeBuildInputs = [ autoPatchelfHook ];

  autoPatchelfIgnoreMissingDeps = [
    "libcrypto.so.10"
    "libssl.so.10"
  ];

  src = fetchurl {
    url = "https://github.com/microsoft/sqltoolsservice/releases/download/${version}/Microsoft.SqlTools.ServiceLayer-rhel-x64-net${dotnet_version}.tar.gz";
    hash = "sha256-hdNZzJLYNO8QKId1q8wqrLLOhE3Wtgmk89vcOD5Ya/A=";
  };

  installPhase = ''
    mkdir -p $out/bin
    tar -x -C $out/bin -f $src
  '';

  postFixup = ''
    patchelf \
      --add-needed libicui18n.so \
      --add-needed libicuuc.so \
      $out/bin/libcoreclr.so \
      $out/bin/*System.Globalization.Native.so
    patchelf --replace-needed liblttng-ust.so.0 liblttng-ust.so $out/bin/libcoreclrtraceptprovider.so
    patchelf \
      --add-needed libgssapi_krb5.so \
      $out/bin/*System.Net.Security.Native.so
    patchelf --add-needed libssl.so \
             $out/bin/*System.Security.Cryptography.Native.OpenSsl.so
  '';

  meta = {
    description = "Microsoft SQL Tools API service that provides SQL Server data management capabilities";
    license = lib.licenses.mit;
  };
}
5 Likes
2

nice work!

What do you think the chances are of actually ‘building’ this package from source, rather than patching the linker paths in prebuilt shared objects?

The source seems to be there and it is ‘built by something’.

Once you nixify the build process you can an PR there and add a shell-nix or a flake.nix to their repo.

everyone on earth needs a reproducible build system…and Microsoft is no exception.

Stirling work, well done.

1 Like
3

Thank you! That is a great idea; if I understand you correctly, you are suggesting I submit a PR to the open-source sqltoolsservice repo, not nixpkgs. Makes sense.

Building from source is a good idea. I admit I was a bit intimidated by someone else’s previous attempt; it seemed quite complex. But it would be worth a try.

I notice that in nixpkgs, there are a variety of approaches; sometimes patching prebuilt binaries with patchelf, even when the code is entirely open-source, while in other instances, the package is built from source. Thoughts on when to choose one approach over another?

4

Does anyone know the reasoning why autoPatchelfHook does not include the path to glibc in the rpath of the patched binaries, while compiling packages from source usually does?

This has now been an issue several times for me and often causes me to fall back to invoking patchelf myself manually.

5

Good question! and welcome to the wonderful world of nix!

It’s certainly worth a try, depending on how unreproduable a projects build system is , however there is a great deal of help available.

Lots of prior art can be found in nix code online, and nixxers who will lend a hand if you do some of the initial ground work yourself.

Nixel (the nix expressional langauge) and the Nix Daemon are a way of describing reproducable builds. However it’s at the whim of the project build systems, some of which are highly unreproducable nature.

We call this the shifting sands problem, you look out, and you see the pyramids, you look out again, and its sand, and then all of sudden the pyramids are back.

Nix builds are concrete not sand.

Patching in nixpkgs, is usually done if the source is not available or the upstream build system is too unreproduable to be nixxified. , Political reasons are something to watch out for too… upstream licensing can be a mine field. At the end of the day nix tries to be a ‘policy free package manager/build systems’.

As i started to study nix, and study licensing i soon found it a very very very complex area, which in some cases can be counter productive to developers ‘getting the job done effectively’ .

Nix likes to eat source code, give to much binary patching and it will get indigestion

However , if it’s a source build, there is no reason why you can’t build it yourself, hell you can even run hydra if you want to start ‘building’ your own cache.nixos.org ,its fun , but my idea of fun and yours may be very different. :wink:

The full list of policies can be found in the nixpkgs contribution guide.

However , sometimes it best to put a ‘draft’ PR in, and someone will leap in and give and give you a hand.

However, nixpkgs is not the only way to, you could create a flake which compiles all this stuff, and submit a flake.nix to the upstream project or submit the flake to the many online flake ‘registries’ that are spring up all over the internet right now.

I found many upstream projects welcome a .nix file into their projects, some even start using nix to build, test and release their software !

On your side, you can set up CI/CD to build that flake repo continuously as the project release new versions. From a small .nix file you get a lot of utility… and sometimes i feel it cannot be possible… but it is.

So, you have options and paths.

Nix really can build anything, if you gave it some bricks it would build a house, and you can build it yourself on your machine, not depending on ‘a cloud service’ to do it for you.

if you gave me the nix file, i could build that house too

nix needs your input and your contribution… it’s not perfect, but it’s getting there… get your friends on board…

if you want a world where ‘it works on my machine’ is no longer a valid response… , which i think is as close ‘developer abusive’ as you can get… (this is a joke) … then keep nixing.

The ecosystem is big, and there are lots of things to learn and get into. The best thing i can recommend is seek out a user group or starting one yourself in your region.

The best way to learn nix is be in a room of nixxers, because sharing is what we do best. https://nix.ug or check discourse for a event near you.

Good luck

and may the nix be with you, always.

1 Like
6

@polygon I have not observed this. If I include stdenv.cc.cc in the buildInputs, then it sure seems like autoPatchelfHook takes care of it.

You must be experiencing different behavior, though? Do you have a reproducible example?