`nix-build` fails because Python wants something that's unavailable, without saying what it wants

At least I think that’s the problem.

I first posted this issue on Stackoverflow. (Nobody responded.) I’ve duplicated the entire text of the post here.

The error

When I run nix-build -A serialosc from the root of nixpkgs, nix prints the following:

[jeff@jbb-dell:~/nix/nixpkgs-fork]$ nix-build -A serialosc
these derivations will be built:
building '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv'...
unpacking sources
unpacking source archive /nix/store/5zj802wfjd0ima92lpzzsqdjqvrnrwf9-serialosc
source root is serialosc
patching sources
configure flags: --prefix=/nix/store/b6qzvzmwjdi29nnzc769904wc0mj1cds-serialosc configure
Setting top to                           : /build/serialosc 
Setting out to                           : /build/serialosc/build 

Checking for 'gcc' (C compiler)          : gcc 

Checking for working poll()              : yes 
Checking for libudev                     : yes 
Checking for libmonome                   : yes 
Checking for liblo                       : yes 
Checking for function strdup             : yes 
Checking for function _strdup            : not found 
Checking for function strndup            : yes 
Checking for function strcasecmp         : yes 
Checking for header unistd.h             : yes 
Checking for header dns_sd.h             : yes 
Checking for library dl                  : yes 

Traceback (most recent call last):
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 103, in waf_entry_point
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 164, in run_commands
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 155, in run_command
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Configure.py", line 92, in execute
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Context.py", line 92, in execute
  File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Context.py", line 133, in recurse
  File "/build/serialosc/wscript", line 259, in configure
  File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
builder for '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv' failed with exit code 2
error: build of '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv' failed


My suspicion

The problem appears to arise in lib/python2.7/subprocess.py. I found a version of that file online, and the definition of _execute_child seems to make reference to some absolute paths – for instance, in this passage:

    if shell:
        args = ["/bin/sh", "-c"] + args
        if executable:
            args[0] = executable

Therefore my guess is that Nix is building in a sandbox that doesn’t offer whatever Python expects. But the error doesn’t tell me what Python was hoping to find.

The context

(You might not need this information.)

I’m trying to build serialosc on NixOS 19.09. Here’s my fork of nixpkgs, including my broken recipe for serialosc. The fork also includes my libmonome package, which works, and which serialosc depends on, and which is not yet part of the official nixpkgs repo. I’ve given hard links to the specific commit; they shouldn’t change when I update the fork.


First, let’s help with the fetchgit part in your recipe

  src = fetchgit {
    # Once, it seemed to finish, and I don't know how! Then it crashed with:
    # Switched to a new branch 'fetchgit'
    # removing `.git'...
    # hash mismatch in fixed-output derivation '/nix/store/5zj802wfjd0ima92lpzzsqdjqvrnrwf9-serialosc':
    # wanted: sha256:1vqcxi32wc4pklbddllflkaigkfvd4ykwrjqccayvrk10dx1sna3
    # got:    sha256:1zmzjasv21ix7i7s58a31k0025ji32hv2jm2ww6s0xhjmr5ax34j

    # This way it [[gets pretty far]].
    # The fetchSubmodules value I set again seems to have no effect.
    url = https://github.com/monome/serialosc.git;
    rev = "v1.4.1";
    sha256 = "1zmzjasv21ix7i7s58a31k0025ji32hv2jm2ww6s0xhjmr5ax34j";

Perhaps there was a force push? Not sure why it would stop working with the hash

I found that the fetchSubmodules has no effect if you have the hash of the one without the submodules, use nix-prefetch-git --fetch-submodules https://github.com/monome/serialosc which gives

  "url": "https://github.com/monome/serialosc",
  "rev": "cec0ea76b2d5f69afa74d3ffc14a0950e32a7914",
  "date": "2019-06-09T21:46:13+02:00",
  "sha256": "03qkzslhih72idwafgfxmkwp5v3x048njh0c682phw2ks11plmbp",
  "fetchSubmodules": true

so use

fetchgit {
  url = https://github.com/monome/serialosc;
  rev = "cec0ea76b2d5f69afa74d3ffc14a0950e32a7914";
  # or rev = version; but that might break with force pushes
  sha256 = "03qkzslhih72idwafgfxmkwp5v3x048njh0c682phw2ks11plmbp";
  fetchSubmodules = true;


There is a (slight) wafHook documentation in NixOS - Nixpkgs 21.05 manual

Debugging error

It seems to want to do git stuff, here is how I debugged it:

$ nix-shell https://github.com/JeffreyBenjaminBrown/nixpkgs/archive/2bbd131811991bec9ae6b296b5cd1441640c306c.tar.gz -A serialosc --pure
nix-shell $ mkdir /tmp/delme-serialosc
nix-shell $ cd !$
# To find out what a build would do
nix-shell $ type genericBuild
# Doing the `if [ -z "${phases:-}" ]; then ...` bit gives us `echo $phases`
# unpackPhase patchPhase configurePhase buildPhase checkPhase installPhase fixupPhase installCheckPhase distPhase
nix-shell $ ${unpackPhase:-unpackPhase}
nix-shell $ cd $sourceRoot
nix-shell $ ${patchPhase:-patchPhase}
nix-shell $ ${configurePhase:-configurePhase}
# Error happens here
nix-shell $ type ${configurePhase:-configurePhase}
# Do the steps until you find it stops in `python ...`
# Python debug info: https://docs.python.org/3/library/pdb.html
nix-shell $ python -m pdb "$wafPath" "${flagsArray[@]}";
(Pdb) continue # or cont, or just c
(Pdb)  break /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py:1047
(Pdb) continue
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(1047)_execute_child()
-> raise child_exception
(Pdb) up 
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(394)__init__()
-> errread, errwrite)
(Pdb) # empty is equivalent to do last command, i.e. up in this case
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(216)check_output()
-> process = Popen(stdout=PIPE, *popenargs, **kwargs)
(Pdb) print(popenargs)
(['git', 'rev-parse', '--verify', '--short', 'HEAD'],)

But you will find you might need to figure a way to avoid waf using git as it cannot access the network unless it’s a fixed-output derivation (e.g. those fetching sources). Hopefully the bit about fetchSubmodules will fix it!

Edit: I didn’t read past that it tries to use git, it tries to figure out what version it is, you should somehow make it avoid using git to do that, e.g. grep -r 'rev-parse' suggests editing the wscript file

Edit 2: E.g. something like

--- wscript-old 2020-01-29 15:55:54.100976818 +0000
+++ wscript     2020-01-29 15:55:58.128977024 +0000
@@ -257,7 +257,7 @@
                conf.env.GIT_COMMIT = subprocess.check_output(
                        ["git", "rev-parse", "--verify", "--short", "HEAD"],
-       except subprocess.CalledProcessError:
+       except (subprocess.CalledProcessError, OSError):
                conf.env.GIT_COMMIT = ''
        conf.define("VERSION", VERSION)

perhaps it could be upstreamed too (this leaves the git commit ID as empty in this case)? Or just use substituteInPlace or something to put in the actual commit ID to be

--- wscript-old 2020-01-29 15:55:54.100976818 +0000
+++ wscript     2020-01-29 15:57:56.177983054 +0000
@@ -249,16 +249,7 @@
        conf.env.VERSION = VERSION
-       try:
-               import os
-               devnull = open(os.devnull, 'w')
-               conf.env.GIT_COMMIT = subprocess.check_output(
-                       ["git", "rev-parse", "--verify", "--short", "HEAD"],
-                       stderr=devnull).decode().strip()
-       except subprocess.CalledProcessError:
-               conf.env.GIT_COMMIT = ''
+        conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'
        conf.define("VERSION", VERSION)
        conf.define("_GNU_SOURCE", 1)
1 Like

That was like drinking from a firehose. I just (over the last four hours) learned a lot. Thanks, @KoviRobi!

And what you suggested works! If I replace those few lines in the wscript with one that hard-codes the git commit, the thing builds. Now I just have to figure out how to encode that replacement in Nix, rather than performing it by hand in the nix shell.

I’ve placed that question last. Before I get to that question, I have a few others:

How did you find this URL?

nix-shell https://github.com/JeffreyBenjaminBrown/nixpkgs/archive/2bbd131811991bec9ae6b296b5cd1441640c306c.tar.gz -A serialosc --pure

(When I run nix-shell, I’m linking to a local folder, not the URL. But being able to find such URLs seems handy.)

Bracket notation in the build phrase list?

When I run type genericBuild, I see this list of phases:

phases="${prePhases:-} unpackPhase patchPhase ${preConfigurePhases:-} configurePhase ${preBuildPhases:-} buildPhase checkPhase  ${preInstallPhases:-} installPhase ${preFixupPhases:-} fixupPhase installCheckPhase  ${preDistPhases:-} distPhase ${postPhases:-}";

In your response you list the phases, but you omit the bracketed expressions. Why? And what do the bracketed expressions mean?

I don’t see the patchPhase step doing anything. Is that normal?

Here’s what it looks like when I run that:

[nix-nshell:/tmp/delme-serialosc/serialosc]$ ${patchPhase:-patchPhase}


How do I replace multiple lines?

Every example I’ve found of substituteInPlace (I looked through dozens) in nixpkgs uses it to replace single-line expressinos. Better yet, is there a way to numerically specify a range of lines?

The nixpkgs documentation for substitute (and related functions) says “you don’t have to worry about escaping special characters”, but that’s absurd – at the least, I’ll have to escape the symbols (") and (’’).

I tried a goofy hack, performing line-by-line substitutions as follows:

  patchPhase = ''
    substituteInPlace wscript
      --replace "conf.env.GIT_COMMIT = subprocess.check_output("
                "conf.env.GIT_COMMIT = \'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914\'"
     --replace "[\"git\", \"rev-parse\", \"--verify\", \"--short\", \"HEAD\"]," ""
     --replace "stderr=devnull).decode().strip()" ""
     --replace "conf.env.GIT_COMMIT = \'\'"
               "conf.env.GIT_COMMIT = cec0ea76b2d5f69afa74d3ffc14a0950e32a7914"

with which I had hoped to effect the following change:

		import os

		devnull = open(os.devnull, 'w')

-		conf.env.GIT_COMMIT = subprocess.check_output(
+		conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'
-			["git", "rev-parse", "--verify", "--short", "HEAD"],
-			stderr=devnull).decode().strip()
	except subprocess.CalledProcessError:
-		conf.env.GIT_COMMIT = ''
+		conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'

When I try that, I get this response:

[nix-shell:/tmp/delme-serialosc/serialosc-cec0ea7]$ ${patchPhase:-patchPhase}
substituteStream(): WARNING: pattern '"conf.env.GIT_COMMIT' doesn't match anything in file 'wscript'
substituteStream(): ERROR: Invalid command line argument: subprocess.check_output("

and after that, wscript is an empty file.

Glad it’s useful, I was trying to get the whole reasoning/process. Firehose v2 follows :wink:

How did you find this URL?

Yes, nix-shell (and nix-env with nix-env -f <url>, as well as nix-build/nix build) support URLs, specifically tarballs, which is indeed pretty handy.

The URL is just a tar.gz archive of your github repo, so you do

and change the extension from .zip to .tar.gz (which GitHub supports, presumably this is the bit you didn’t know, I think they just use zip by default to avoid confusing non-Unix people).

Bracket notation in the build phrase list?

This will be an introduction to programming in sh/bash, for more details I refer you to the bash manual page, e.g. type in man bash into your terminal.

Bash is a slightly odd programming language, in that everything is quoted as a string, unless you use $ to get the value of that variable, e.g. foo is just a string just like "foo", but $foo is the value of the variable. Note, that $ expands variables inside double-quoted strings, but not single-quoted strings, so "$foo" is the value of the variable foo, while '$foo' is the string dollar foo.

In some cases, you want the value of the variable foo followed immediately by a string, say bar, so you use the curly brackets/braces, e.g. ${foo}bar, as $foobar is the value of the variable called foobar.

On top of this, bash supports some replacements, one of which is the “use this value if the variable is unset or empty”, which is the colon-minus ${prePhases:-} which says use the empty string if the variable prePhases is unset or empty.

Now, normally a variable that is unset is treated as the empty string, but for the sake of catching errors earlier, the builder is run with flags e/errexit (exit at errors) u/nounset (unset variables are an error), this is done by the file $stdenv/setup with set -eu.

Because of this, we use ${prePhases:-} instead of just ${prePhases}.

I ignored the bracketed expressions, because they are unset in your derivation. To set them, you could do in your default.nix something like

stdenv.mkDerivation rec {
   ... # Other stuff
   prePhases = ''
   ... # Other stuff

(the stdenv.mkDerivation is just for context as to where to put the prePhases).

I don’t see the patchPhase step doing anything. Is that normal?

Yes it’s normal for the patchPhase to do nothing, unless you have defined patches in your nix derivation.

But let me use this space for Bash tutorial part 2; what does ${patchPhase:-patchPhase} mean?

Well, it gets the value of the variable patchPhase, or if that is unset/empty, it uses the string patchPhase. But what is the string patchPhase used for then? Well, you can think of bash as having two stages, variable expansion, and evaluation. So once the command line has had variable expansion, say the variable patchPhase is unset/empty, we evaluate the command line patchPhase, for which bash searches for in the builtins, the defined functions, and the executable files in the colon-separated list of directories of the variable $PATH.

To see where it finds the patchPhase, you can use type patchPhase, or to see all matches, use type -a patchPhase. In this case it’s a function (that I know to be defined in the file $stdenv/setup.

How do I replace multiple lines?

Now, you might have the question, how do I stop/override a phase, or what went wrong with the patchPhase I tried?

Well, you were on the right track actually, you just need to tell bash that you are entering a multi-line expression, which is what the backslashes \ at the end of the line do. Also, you don’t need to escape a single-quote inside a double-quoted string. But this leads to having two single-quotes, which terminates the nix string. You can escape those with another single-quote. (Note, there is nix shell if you want to play around with nix expressions.)

  patchPhase = ''
    substituteInPlace wscript                                                       \
      --replace "conf.env.GIT_COMMIT = subprocess.check_output("                    \
                "conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'"  \
     --replace "[\"git\", \"rev-parse\", \"--verify\", \"--short\", \"HEAD\"]," ""  \
     --replace "stderr=devnull).decode().strip()" ""                                \
     --replace "conf.env.GIT_COMMIT = '''"                                          \
               "conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'"

The backslashes need not be aligned, but it helps visually. They must however be the last thing on the line, not even space after them, as you are escaping the newline character.

As for disabling a phase, remember how I said the :- is for unset or empty, well there are nix variables for the mkDerivation such as dontConfigure or doCheck, but for the sake of some more bash, you could do something like configurePhase = ":" as in bash the colon (:) is the do nothing operation. Also, instead of :- for unset or empty, there is - for only unset, e.g. we could have used ${patchPhase-patchPhase} which when the environment variable patchPhase is empty, evaluates to empty.

As a side-note, yes, the bash variable expansion rules are unfortunately complex/subtle, and it’s one of those things that can lead to hard-to-find bugs.

As for how to do the substitution, I would just create a patch, say 0001-fix-wscript-lack-of-git.patch containing

--- a/wscript 2020-01-30 17:27:13.264537220 +0000
+++ b/wscript 2020-01-30 17:35:56.911563968 +0000
@@ -257,8 +257,9 @@
                conf.env.GIT_COMMIT = subprocess.check_output(
                        ["git", "rev-parse", "--verify", "--short", "HEAD"],
-       except subprocess.CalledProcessError:
-               conf.env.GIT_COMMIT = ''
+       except (OSError, subprocess.CalledProcessError):
+                import os
+               conf.env.GIT_COMMIT = os.getenv("version")
        conf.define("VERSION", VERSION)
        conf.define("_GNU_SOURCE", 1)

Note, that tabs inevitably got lost here, where as the original wscript file had tabs, so you might have to recreate the patch on your machine. See here for patch creation details.

With the patch, it almost builds for me, except it fails for some pointer alignment reason

../third-party/libuv/src/unix/linux-core.c: In function 'uv__io_poll':
../third-party/libuv/src/unix/linux-core.c:268:5: error: converting a packed 'struct uv__epoll_event' pointer (alignment 1) to a 'uv__io_t' {aka 'struct uv__io_s'} pointer (alignment 8) may result in an unaligned pointer value [-Werror=address-of-packed-member]
  268 |     loop->watchers[loop->nwatchers] = (void*) events;
      |     ^~~~
In file included from ../third-party/libuv/src/unix/internal.h:37,
                 from ../third-party/libuv/src/unix/linux-core.c:22:
../third-party/libuv/src/unix/linux-syscalls.h:97:8: note: defined here
   97 | struct uv__epoll_event {
      |        ^~~~~~~~~~~~~~~
In file included from /build/serialosc/third-party/libuv/include/uv.h:61,
                 from ../third-party/libuv/src/unix/linux-core.c:21:
/build/serialosc/third-party/libuv/include/uv-unix.h:84:8: note: defined here
   84 | struct uv__io_s {
      |        ^~~~~~~~
cc1: all warnings being treated as errors

Waf: Leaving directory `/build/serialosc/build'
Build failed
 -> task in 'LIBUV' failed (exit status 1):
        {task 140737336483152: c linux-core.c -> linux-core.c.1.o}
['gcc', '-std=c99', '-Wall', '-Werror', '-pthread', '-g', '--std=gnu89', '-pedantic', '-Wall', '-Wextra', '-Wno-unused-parameter', '-Wstrict-aliasing', '-fPIC', '-Wno-cast-align', '-I/build/serialosc/build/third-party/libuv/include', '-I/build/serialosc/third-party/libuv/include', '-I/build/serialosc/build/third-party/libuv/src', '-I/build/serialosc/third-party/libuv/src', '-DHAVE_WORKING_POLL=1', '-DHAVE_LIBUDEV=1', '-DHAVE_LIBMONOME=1', '-DHAVE_LO=1', '-DHAVE_STRDUP=1', '-DHAVE_STRNDUP=1', '-DHAVE_STRCASECMP=1', '-DHAVE_UNISTD_H=1', '-DHAVE_DNS_SD_H=1', '-DVERSION="1.4.1"', '-D_GNU_SOURCE=1', '-DGIT_COMMIT="v1.4.1"', '-D_LARGEFILE_SOURCE', '-D_FILE_OFFSET_BITS=64', '-D_GNU_SOURCE', '-D_LARGEFILE_SOURCE', '-D_FILE_OFFSET_BITS=64', '-D_POSIX_C_SOURCE=200112', '../third-party/libuv/src/unix/linux-core.c', '-c', '-o', '/build/serialosc/build/third-party/libuv/src/unix/linux-core.c.1.o']
note: keeping build directory '/tmp/nix-build-serialosc.drv-2'
builder for '/nix/store/01g9ikl18plly7big7ajkih8kr0n5zgi-serialosc.drv' failed with exit code 1
error: build of '/nix/store/01g9ikl18plly7big7ajkih8kr0n5zgi-serialosc.drv' failed

You’d have to ask serialosc’s developers about this issue.

1 Like