Faking root in build sandbox

sheepforce · January 11, 2022, 11:16pm

Hey
I am trying to write a derivation for a proprietary image processing program called PixInsight. It has a complex runtime (many X11 libraries, libGL, a lot of network, audio, video libs). Next to huge amounts of binaries, there are also many Javascript and bash files and I basically gave up on patching all binaries and scripts and I am OK with using buildFHSUserEnv. However, there is an installer (another C++ binary), which does a lot of magic. I wasn’t successfull in reverse engineering everything it does and can therefore not replicate its behaviour. Not running the installer properly, leads to no usable installation/directory structure. Unfortunately the installer strictly checks for being executed by root, as it installs into /opt, /bin, /usr/share/ by default. All these paths can be changed (and I want to put them into $out) by command line options, but the requirement for root cannot be circumvented. I’ve tried to fake root by calling unshare in the installPhase, but this does not seem to be possible. I’ve also tried proot with no success. It is of course possible that I am doing something wrong, though.

Does anyone have an idea how to execute a binary installer, which cecks for being called by root, in the build sandbox?

Best wishes
Phillip

bartsch · January 12, 2022, 9:02am

You could try to figure out what the installer is checking by using ltrace.

If it is using a library call (i.e. getuid()) to check for root, then you could write a shared library, which you LD_PRELOAD, to change the return value of the library call. Example: https://www.sweharris.org/post/2017-03-05-ld-preload/

L-as · January 12, 2022, 12:45pm

bubblewrap is your friend

t184256 · January 13, 2022, 9:25am

Nah, if unshare didn’t work, how’d bubblewrap help?

Granted, the topic starter isn’t exactly generous with details of how things didn’t work.

sheepforce · January 13, 2022, 10:20am

I didn’t have the time to try the ideas, yet, but you are right, i was a little bit spare with details. Here they are:

This is what I’ve tried so far (the installPhase just asks for ./installer --help, but the real arguments don’t change anything):

{ stdenv, lib, requireFile, buildFHSUserEnv, breakpointHook, wrapQtAppsHook, util-linux, proot }:

let
  piStore = stdenv.mkDerivation rec {
    pname = "pixinsight";
    version = "1.8.8-12";

    src = requireFile rec {
      name = "PI-linux-x64-${version}-20211229-c.tar.xz";
      url = "https://pixinsight.com/";
      sha256 = "7095b83a276f8000c9fe50caadab4bf22a248a880e77b63e0463ad8d5469f617";
    };

    nativeBuildInputs = [ util-linux proot wrapQtAppsHook breakpointHook ];
    dontWrapQtApps = true;

    sourceRoot = ".";

    postPatch = ''
      patchelf ./installer \
        --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
        --set-rpath ${stdenv.cc.cc.lib}/lib
    '';

    dontConfigure = true;
    dontBuild = true;

    installPhase = ''
      runHook preInstall

      mkdir -p $out
      unshare -r ./installer --help

      runHook postInstall
    '';

    postFixup = ''
      wrapQtApp $out/bin/PixInsight
    '';

  };
in buildFHSUserEnv {
  name = "pixinsight";

  targetPkgs = pkgs: (with pkgs; [
    libGL
    libpulseaudio
    alsa-lib
    nss
    gd
    gst_all_1.gstreamer
    nspr
    expat
    fontconfig
    dbus
    glib
    zlib
    piStore
  ]) ++ (with pkgs.xorg; [
    libX11
    libXdamage
    xrandr
    libXtst
    libXcomposite
    libXext
    libXfixes
    libXrandr
  ]);

  profile = ''
    export LD_LIBRARY_PATH=${piStore}/bin/lib:$LD_LIBRARY_PATH
    export LC_ALL=en_US.utf8
    export QT_PLUGIN_PATH=${piStore}/bin/lib/qt-plugins
    export QT_QPA_PLATFORM_PLUGIN_PATH=${piStore}/bin/lib/qt-plugins/platforms
    export QT_QPA_PLATFORMTHEME=
    export QT_QPA_GENERIC_PLUGINS=
    export QT_AUTO_SCREEN_SCALE_FACTOR=0
    export QT_ENABLE_HIGHDPI_SCALING=0
    export QT_SCALE_FACTOR=1
    export QT_LOGGING_RULES='*=false'
    export QTWEBENGINEPROCESS_PATH=${piStore}/bin/libexec/QtWebEngineProcess
    export AVAHI_COMPAT_NOWARN=1
  '';

  runScript = "${piStore}/bin/PixInsight";
}

To my surprise this fails with no error message in the installPhase:

building '/nix/store/hir661djlvcd9x3zm4cw7psfj3znny9i-pixinsight-1.8.8-12.drv'...
qtPreHook
unpacking sources
unpacking source archive /nix/store/p5n8ziabj88p9dxwnwml7yn778w77rqh-PI-linux-x64-1.8.8-12-20211229-c.tar.xz
source root is .
setting SOURCE_DATE_EPOCH to timestamp 1640818872 of file ./installer
patching sources
glibPreInstallPhase
installing
build failed in installPhase with exit code 1

I am then attaching with cntr:

> cntr attach -t command cntr-/nix/store/i93i6mz9k60iscii9r4vkmmxcrq83fy0-pixinsight-1.8.8-12
> cd build 
> unshare -r ./installer --help
unshare: unshare failed: Operation not permitted

Alternatively with proot -0 ./installer --help nix-build fails the same way. However, the proot command is not available when attaching with cntr.

The idea with LD_PRELOAD is very interesting. I am not very experienced with C/C++ and ltrace, but ltrace ./installer |& grep uid actually gives a getuid() = 9038, so this might actually be an alternative option.

Atemu · January 13, 2022, 2:46pm

Hack: Find the UID comparison conditional in the binary and patch it into a tautology.

A better solution is to get fakeroot working. What’s the error?

sheepforce · January 13, 2022, 3:20pm

Hu, i didn’t know about fakeroot. I’ve changed from my original version as follows:

  nativeBuildInputs = [ fakeroot ];
  installPhase = ''
      mkdir -p $out/share/{applications,mime,icons/hicolor}

      fakeroot ./installer \
        --yes \
        --install-dir=$out \
        --install-desktop-dir=$out/share/applications \
        --install-mime-dir=$out/share/mime \
        --install-icons-dir=$out/share/icons/hicolor \
        --no-bin-launcher
  '';

But similar to the proot and unshare versions it does not print any error at all, just exit code 1

building '/nix/store/z533li19vkiq26p3xmksw224l7v6y7fl-pixinsight-1.8.8-12.drv'...
qtPreHook
unpacking sources
unpacking source archive /nix/store/p5n8ziabj88p9dxwnwml7yn778w77rqh-PI-linux-x64-1.8.8-12-20211229-c.tar.xz
source root is .
setting SOURCE_DATE_EPOCH to timestamp 1640818872 of file ./installer
patching sources
glibPreInstallPhase
installing
build failed in installPhase with exit code 1

bartsch · January 14, 2022, 2:16pm

strace said, that the installer is trying to get tty setting (ioctl with TCGETS), which fails as there’s no tty. script from util-linux can solve the problem:

fakeroot script -ec "./installer --yes ..." /dev/stdout

Then the installer starts. Though, I had to adjust things to make it work:

–install-dir=$out will wipe $out, I changed it to $out/opt aswell as the paths to bin/PixInsight
$out/share/mime needs another packages sub-directory.

To be able to execute PixInsight, it needs more dependencies, i.e. openssl and Qt libraries (just used qt5.full). I’ve stopped at cudatoolkit.

I hope this helps you.

sheepforce · January 14, 2022, 4:57pm

Ah, this was the final tip to make it work in addition to the fakeroot stuff. Thank you so much. This is the fully working expression:

{ stdenv, lib, requireFile, buildFHSUserEnv, breakpointHook, wrapQtAppsHook, autoPatchelfHook
, makeWrapper, unixtools, fakeroot, mime-types, libGL, libpulseaudio, alsa-lib, nss, gd, gst_all_1
, nspr, expat, fontconfig, dbus, glib, zlib, openssl, libdrm, cups, avahi-compat, xorg, wayland
, libudev0-shim
# Qt 5 subpackages
, qt5
# libsForQt5 sub packages.
, qt3d, mlt
}:

stdenv.mkDerivation rec {
  pname = "pixinsight";
  version = "1.8.8-12";

  src = requireFile rec {
    name = "PI-linux-x64-${version}-20211229-c.tar.xz";
    url = "https://pixinsight.com/";
    sha256 = "7095b83a276f8000c9fe50caadab4bf22a248a880e77b63e0463ad8d5469f617";
  };
  sourceRoot = ".";

  nativeBuildInputs = [
    unixtools.script
    fakeroot
    wrapQtAppsHook
    autoPatchelfHook
    breakpointHook
    mime-types
    libudev0-shim
  ];

  buildInputs = [
    stdenv.cc.cc.lib
    stdenv.cc
    libGL
    libpulseaudio
    alsa-lib
    nss
    gd
    gst_all_1.gstreamer
    gst_all_1.gst-plugins-base
    nspr
    expat
    fontconfig
    dbus
    glib
    zlib
    openssl
    libdrm
    wayland
    cups
    avahi-compat
    # Qt stuff
    qt3d
    mlt
  ] ++ (with xorg; [
    libX11
    libXdamage
    xrandr
    libXtst
    libXcomposite
    libXext
    libXfixes
    libXrandr
  ]) ++ (with qt5; [
    qtbase
    qtgamepad
    qtserialport
    qtserialbus
    qtvirtualkeyboard
    qtmultimedia
    qtwebkit
  ]);

  postPatch = ''
    patchelf ./installer \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath ${stdenv.cc.cc.lib}/lib
  '';

  dontConfigure = true;
  dontBuild = true;

  installPhase = ''
    # mkdir -p $out/share/{applications,mime/packages,icons/hicolor}
    mkdir -p $out/bin $out/opt/PixInsight $out/share/{applications,mime/packages,icons/hicolor}

    fakeroot script -ec "./installer \
      --yes \
      --install-dir=$out/opt/PixInsight \
      --install-desktop-dir=$out/share/applications \
      --install-mime-dir=$out/share/mime \
      --install-icons-dir=$out/share/icons/hicolor \
      --no-bin-launcher \
      --no-remove"

    rm -rf $out/opt/PixInsight-old-0
    ln -s $out/opt/PixInsight/bin/PixInsight $out/bin/.
  '';

  # Some very exotic Qt libraries are not available in nixpkgs
  autoPatchelfIgnoreMissingDeps = true;

  # This mimics what is happening in PixInsight.sh and adds on top the libudev0-shim, which
  # without PixInsight crashes at startup.
  qtWrapperArgs = [
    "--prefix LD_LIBRARY_PATH : ${libudev0-shim}/lib"
    "--set LC_ALL en_US.utf8"
    "--set AVAHI_COMPAT_NOWARN 1"
    "--set QT_PLUGIN_PATH $out/opt/PixInsight/bin/lib/qt-plugins"
    "--set QT_QPA_PLATFORM_PLUGIN_PATH $out/opt/PixInsight/bin/lib/qt-plugins/platforms"
    "--set QT_AUTO_SCREEN_SCALE_FACTOR 0"
    "--set QT_ENABLE_HIGHDPI_SCALING 0"
    "--set QT_SCALE_FACTOR 1"
    "--set QT_LOGGING_RULES '*=false'"
    "--set QTWEBENGINEPROCESS_PATH $out/opt/PixInsight/bin/libexec/QtWebEngineProcess"
  ];
  dontWrapQtApps = true;
  postFixup = ''
    wrapProgram $out/opt/PixInsight/bin/PixInsight ${builtins.toString qtWrapperArgs}
  '';
}

Atemu · January 15, 2022, 11:47am

Awesome! Could you add a meta field and submit it upstream for assimilation into Nixpkgs?

L-as · January 15, 2022, 12:38pm

Your kernel likely disables unprivileged user namespaces. You should probably fix this rather than working around it.

ElvishJerricco · January 15, 2022, 3:10pm

You could also have just used runInLinuxVM. It’s very slightly slower since it has to boot up nixos, and it requires you to have linux Nix build machines that support KVM. But it would work quite well if these aren’t issues for you.

Basically it starts a VM, and replaces init with a script that calls does a bit of setup and calls your builder. All using a virtfs file system, so it doesn’t require huge disk image.

payas · January 15, 2022, 4:15pm

How does that affect interaction with host OS? E.g. reading/writing files, (if required) network access, clipboard access and so on…

ElvishJerricco · January 15, 2022, 4:17pm

Because the root fs in the VM is writable IIRC, and your builder gets run as root. So you do your root stuff and move it to $out, which corresponds to the actual out on the host.

Oh but network access is still not available.

sheepforce · January 15, 2022, 11:22pm

Sure, can do. I am not so sure how helpful this is to others, as this is proprietary software and quite specific to astrophotography, but if you think it is useful, I will open a PR and ping you for the review.

sheepforce · January 15, 2022, 11:36pm

Namespaces actually work fine. I can execute unshare properly in a normal shell. Meanwhile I figured out that it actually also works in the build sandbox. The second problem that @bartsch pointed out (the ioctl call) actually prevented it from working. Interestingly unshare only fails when attaching with cntr to the breakpointHook. I have no idea why, though.

AlexV1 · June 20, 2023, 3:32am

Guess what, I am also using PixInsight and trying to use NixOS as my daily driver.

sheepforce · June 20, 2023, 9:08pm

Oh very nice. I am surprised to find such overlaps in the NixOS community. With @markuskowa I was already very lucky to find someone else doing quantum chemistry

Does PixInsight work nicely for you so far in Nix? I will try to migrate my equipment control PC from Debian to NixOS in July probably and see if INDI in NixOS works nicely. Feel free to shoot me a PM if you like.