How to package python program that calls subprocess.run()?

Hi,
I am trying to package my first program with Nix. It is called wg-netns (GitHub - dadevel/wg-netns: WireGuard with Linux Network Namespaces) and it contains code to call external commands (such as the “ip” command) by calling subprocess.run(). This does not work and I wonder how I can best adjust it to work on NixOS?

From the source code of it:

def ip(*args, stdin: str = None, check=True, capture=False) -> str:
    return run('ip', *args, stdin=stdin, check=check, capture=capture)


def host_eval(*args, stdin: str = None, check=True, capture=False) -> str:
    return run(SHELL, '-c', *args, stdin=stdin, check=check, capture=capture)


def run(*args, stdin: str = None, check=True, capture=False) -> str:
    args = [str(item) if item is not None else '' for item in args]
    if VERBOSE:
        print('>', ' '.join(args), file=sys.stderr)
    process = subprocess.run(args, input=stdin, text=True, capture_output=capture)
    if check and process.returncode != 0:
        error = process.stderr.strip() if process.stderr else f'exit code {process.returncode}'
        raise RuntimeError(f'subprocess failed: {" ".join(args)}: {error}')
    return process.stdout

In what sense does it not work?

It tries to call the “ip” command but since it is not in the path it fails.
I forgot to say, that it is called through a systemd unit (https://github.com/dadevel/wg-netns/blob/41665ca136007de5c378703f978085eecde56db0/extras/wg-netns%40.service)

systemctl status wg-netns@mullvad-fi1
× wg-netns@mullvad-fi1.service - WireGuard Network Namespace (mullvad-fi1)
     Loaded: loaded (/etc/systemd/system/wg-netns@.service; disabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-10-18 21:19:54 CEST; 5s ago
    Process: 6027 ExecStart=/nix/store/g58c5k5q0y5rvzkb4qwr99q8c76pgl0k-wg-netns-/bin/wg-netns up mullvad-fi1 (code=exited, status=1/FAILURE)
   Main PID: 6027 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 231ms

okt 18 21:19:54 mediaclient wg-netns[6027]:   File "/nix/store/9srs642k875z3qdk8glapjycncf2pa51-python3-3.10.7/lib/python3.10/subprocess.py", line 501, in run
okt 18 21:19:54 mediaclient wg-netns[6027]:     with Popen(*popenargs, **kwargs) as process:
okt 18 21:19:54 mediaclient wg-netns[6027]:   File "/nix/store/9srs642k875z3qdk8glapjycncf2pa51-python3-3.10.7/lib/python3.10/subprocess.py", line 969, in __init__
okt 18 21:19:54 mediaclient wg-netns[6027]:     self._execute_child(args, executable, preexec_fn, close_fds,
okt 18 21:19:54 mediaclient wg-netns[6027]:   File "/nix/store/9srs642k875z3qdk8glapjycncf2pa51-python3-3.10.7/lib/python3.10/subprocess.py", line 1845, in _execute_child
okt 18 21:19:54 mediaclient wg-netns[6027]:     raise child_exception_type(errno_num, err_msg, err_filename)
okt 18 21:19:54 mediaclient wg-netns[6027]: FileNotFoundError: [Errno 2] No such file or directory: 'ip'
okt 18 21:19:54 mediaclient systemd[1]: wg-netns@mullvad-fi1.service: Main process exited, code=exited, status=1/FAILURE
okt 18 21:19:54 mediaclient systemd[1]: wg-netns@mullvad-fi1.service: Failed with result 'exit-code'.

Then you need to make sure that it is either in the PATH of the unit (systemd.services.<name>.path) or a proper dependency of the python program by wrapping it using wrapProgram.

Thanks for those hints, I will look into them!

If those are a hard requirement for the software to work you must ensure they are available, and preferably not load them impurely from the environment. You have a few options:

  1. Create patches that add @command@ placeholders and the run substituteAll to replace them with the appropriate path during the build.
    (works for every software, not just python modules, but it’s quite a burden to maintain)

  2. Alternatively, use substituteInPlace or sed to do the replacing manually.
    (less annoying but you have to be careful not to mess up the source)

  3. Python programs are always wrapped, so you can exploit this to add the packages you need to the PATH. Add this to the buildPythonPackage arguments:
    makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ ... ]}" ];
    (this is probably the best option)

3 Likes

To expand on what @rnhmjoj said.

If the python code is used as a module (e.g. import <pkg>), then you have to patch

or

If the package is meant to be used as a command (e.g. glances --help), then you can just ask pythonBuild{Application,Package} to wrap it for you:

Thanks for good suggestions. I think I like option 3 best. The program is usually run using the systemd service, but proper packaging needs to handle both ways, I think (using service or invoked directly)

I am trying to do the same but for a python module, i.e. makeWrapperArgs cannot be used. The module calls swig and gfortran. Is there no other way than to patch, in order to make them available? I see that the patches have to be updated each time the module version is bumped.

Can you point to an example in <nixpkgs> where option 1 is used, so I can get an idea of how to do it?

Why? The patches will always apply (with fuzz) unless the very line you’re patching changed. I doubt that would happen on every release.