How to create setuid wrapper for installed program?

scalavision · April 3, 2019, 1:26pm

Hi all,

Singularity is a container solution similar to Docker, much used in HPC environments. Trying to make singularity run without sudo has been challenging on NixOS. It needs setuid bit set in the initial startup from what I understand:

https://www.sylabs.io/guides/3.0/user-guide/security_options.html

Trying to run singularity as a normal user gives this error:

ERROR  : Failed to set effective UID to 0

adding the --debug flag conveys:

ERROR   [U=1000,P=7931]    priv_escalate()               Failed to set effective UID to 0

Reading github PR’s and issues, searching documentation, it’s been pretty hard to find out how to actually add the CAP_SETUID to singularity after its installation.

According to the docs linked above, you could use this command to add a capability:

sudo singularity capability add --user david CAP_NET_RAW

That is of course not possible since the /nix/store is read only. Is there a way to add this capability in the ~/.config/nixpkgs/config.nix ?

Would be happy to add docs to the NixOS manual or Wiki, if I find such a way

makefu · April 3, 2019, 8:02pm

There is security.wrappers which you can use to to create a wrapper for a binary, see NixOS Search
The wrapper can in turn be found in /run/wrappers/bin/<name>

Another option of course is to start the daemon via systemd, where you can either start it as the root user or set the correct Capabilities.

I hope this helps!

scalavision · April 4, 2019, 2:02pm

thank you @makefu !

However, I am sorry, but I don’t understand how I can use this security.wrapper. Can I use it after the package is installed? As I understand, I have to add this to the installation script, default.nix somehow, but how ? I can only see examples using this field for services as you mention or in module . Do I have to write a module for singularity to make this work? I can see something similar done with virtualbox. I am only using singularity to wrap other tools, so it will not run as a service for my usage.

The default.nix has this in its postConfigure, I guess this is just to avoid installing some things that needs setuid in the installation phase:

    # Don't install SUID binaries
    sed -i 's/-m 4755/-m 755/g' builddir/Makefile

Then in the installPhase, they try to set some permissions on something called starter-suid:

chmod 755 $bin/libexec/singularity/bin/starter-suid

I tried adding this to my ~/.config/nixpkgs/config.nix:

security.wrappers = {                                                                                                                       
singularity.sources = "/nix/store/ykcfcmwqwc4qfn4kfxmsxafn8r21jbjb-singularity-3.0.1-bin/singularity";                                    
};

it neither helps or crash when I reinstall the personal profile.

scalavision · April 4, 2019, 2:40pm

Hm… seems they have already created a module for singularity 2 months ago. It used to be possible to run singularity without sudo. Maybe this changed it. I will open an issue …

github.com

NixOS/nixpkgs/blob/b306b4cf1a9709dd8e671af6ebf75151e359e54f/nixos/modules/programs/singularity.nix

{ config, pkgs, lib, ... }:

with lib;
let
  cfg = config.programs.singularity;
  singularity = pkgs.singularity.overrideAttrs (attrs : {
    installPhase = attrs.installPhase + ''
      mv $bin/libexec/singularity/bin/starter-suid $bin/libexec/singularity/bin/starter-suid.orig
      ln -s /run/wrappers/bin/singularity-suid $bin/libexec/singularity/bin/starter-suid
    '';
  });
in {
  options.programs.singularity = {
    enable = mkEnableOption "Singularity";
  };

  config = mkIf cfg.enable {
      environment.systemPackages = [ singularity ];
      security.wrappers.singularity-suid.source = "${singularity}/libexec/singularity/bin/starter-suid.orig";
      systemd.tmpfiles.rules = [

This file has been truncated. show original

jtojnar · April 4, 2019, 2:29pm

Nix cannot create setuid programs, since, in general, non-privileged users could use it to create arbitrary setuid programs. The only way to get setuid program is through NixOS module in configuration.nix. Instead of writing a module you can instruct users to add the option directly to their configuration.nix` but modules are preferred for convenience.

scalavision · April 4, 2019, 2:55pm

thank you @jtojnar, I am a bit confused by setting this option in configuration.nix. Adding singularity here does not seem to work either :

 # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
programs.singularity.enable = true;

This might be because I’ve tried installing multiple instances now, and there may be something wrong with paths etc.

This error seems actually related to these:

Failed to run singularity as non-root user because of the privilege to operate /dev/loop* · Issue #49683 · NixOS/nixpkgs · GitHub
singularity: 2.6.0 -> 3.0.1 by jbedo · Pull Request #50939 · NixOS/nixpkgs · GitHub

I’ll investigate further. Thanks again for your help

jbedo · April 4, 2019, 8:25pm

Enabling the module should take care of everything for you and you shouldn’t need to do any SUID wrapping yourself. Make sure singularity is not in your environment.systemPackages or your user profile though.