Generic way to use nix-shell to compile a project

I use since recently a great feature to compile and debug a project quickly: nix-shell. My goal was to run:

$ nix-shell
[nix-shell]$ genericBuild
... the build runs until errors
[nix-shell]$ # fix the errors, and run only the missing phases with `phases="buildPhase" genericBuild`

Unfortunately, on the project I’m currently working on, the process fails and exit (in the nix-shell) during the unpackPhase, because it tries to unpack a folder… which is already unpacked. Since nix-build do not have any problem during the unpack phase, I guess that the function that I should run is maybe not genericBuild. To check, I tried to find the actual builder, which is bash, but I can’t find any argument:

[nix-shell]$ echo $builder
/nix/store/cwnwyy82wrq53820z6yg7869z8dl5s7g-bash-4.4-p23/bin/bash

[nix-shell]$ echo $args

So how can I know and reproduce the exact commands that nix-build is going to run? Also, any idea why in my case the unpack phase fails in nix-shell but not in nix-build (I’m using fetchSubmodules, don’t know if it’s the reason…)? (and if you understand why my nix-build fails, I’d also be interested, but it’s another question I guess)

Thanks!

derivation.nix:

{ lib, clangStdenv, fetchFromGitHub, nodejs }:

clangStdenv.mkDerivation rec {
  pname = "armorpaint";
  version = "20.01";

  src = fetchFromGitHub {
    owner = "armory3d";
    repo = "armorpaint";
    rev = "95453348b2";
    sha256 = "sha256-wmgEXcPZw5JWnIPzw18lKCu7xMVQ+rFYH0Z93/n8qrU=";
    fetchSubmodules = true;
  };

  buildInputs = [ nodejs ];

  buildPhase = ''
    node armorcore/make -g opengl
    cd armorcore
    node Kinc/make -g opengl --compiler clang --compile
    cd Deployment
    strip Krom
    ./Krom ../../build/krom
  '';

  # meta = with stdenv.lib; {
  #   description = "";
  #   homepage = "https://github.com/armory3d/armorpaint";

  #   license = licenses.;
  #   maintainers = with maintainers; [  ];
  #   platforms = platforms.;
  # };
}

default.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}

Console:

# Console output:
$ nix-shell

[nix-shell]$ genericBuild 
unpacking sources
unpacking source archive /nix/store/68v9kl0824hvwzjl9l8azmdsjjqf6khp-source
unpacker appears to have produced no directories
exit

$ ls /nix/store/68v9kl0824hvwzjl9l8azmdsjjqf6khp-source
armorcore  Assets  changes.md  checkstyle.json  khafile.js  Libraries  LICENSE.md  README.md  Shaders  Sources

nix-build works (at least it unpacks correctly):

$ nix-build 
this derivation will be built:
  /nix/store/6gvb0ysbk1ljwsqxxrrlgvrwbdl13v0w-armorpaint-20.01.drv
building '/nix/store/6gvb0ysbk1ljwsqxxrrlgvrwbdl13v0w-armorpaint-20.01.drv'...
unpacking sources
unpacking source archive /nix/store/68v9kl0824hvwzjl9l8azmdsjjqf6khp-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 1617402671 of file source/khafile.js
patching sources
configuring
no configure script, doing nothing
building
Using Kha from /build/source/armorcore
Creating Kha project.
Exporting asset 1 of 28 (Scene.arm).
Exporting asset 2 of 28 (World_irradiance.arm).
Exporting asset 3 of 28 (World_radiance.hdr).
Exporting asset 4 of 28 (World_radiance_0.hdr).
  [...]
Compiling shader 6 of 50 (armdefault_mesh.frag.glsl).
Compiling shader 7 of 50 (armdefault_mesh.vert.glsl).
events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: spawn /build/source/armorcore/Kinc/Tools/krafix/krafix-linux64 ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
    at onErrorNT (internal/child_process.js:465:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
    at onErrorNT (internal/child_process.js:465:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /build/source/armorcore/Kinc/Tools/krafix/krafix-linux64',
  path: '/build/source/armorcore/Kinc/Tools/krafix/krafix-linux64',
  spawnargs: [
    'glsl',
    '/build/source/Shaders/common/Gizmo_overlay.frag.glsl',
    '/build/source/build/krom/data/Gizmo_overlay.frag.glsl.temp',
    '/build/source/build/temp',
    'krom'
  ]
}
error: --- Error --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
builder for '/nix/store/6gvb0ysbk1ljwsqxxrrlgvrwbdl13v0w-armorpaint-20.01.drv' failed with exit code 1; last 10 log lines:
    syscall: 'spawn /build/source/armorcore/Kinc/Tools/krafix/krafix-linux64',
    path: '/build/source/armorcore/Kinc/Tools/krafix/krafix-linux64',
    spawnargs: [
      'glsl',
      '/build/source/Shaders/common/Gizmo_overlay.frag.glsl',
      '/build/source/build/krom/data/Gizmo_overlay.frag.glsl.temp',
      '/build/source/build/temp',
      'krom'
    ]
  }

I agree it would be cool if this were easier!

You can run type genericBuild to see what ‘genericBuild’ does - perhaps even save that, edit it to skip the unpack phase, and run it?

Well, what I don’t understand is that genericBuild should do exactly the same thing in both nix-shell and nix-build, no? Otherwise wouldn’t it be considered as a bug? I can try to debug to see what is the exact difference, but it could take some time for me to understand…

Even if I run exactly the exact command, I get the error. Using:

nix show-derivation $(nix-instantiate)

I have access to the builder and args (don’t know why nix-shell does not put these values in the environment variables as well): in my case it’s:

/nix/store/cwnwyy82wrq53820z6yg7869z8dl5s7g-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh

But if I run that, I still get the error:

$ nix-shell

[nix-shell]$ /nix/store/cwnwyy82wrq53820z6yg7869z8dl5s7g-bash-4.4-p23/bin/bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
unpacking sources
unpacking source archive /nix/store/68v9kl0824hvwzjl9l8azmdsjjqf6khp-source
unpacker appears to have produced no directories

I don’t understand what’s wrong.

Also, I don’t understand something else: if I do:

$ echo $buildPhase
node armorcore/make -g opengl cd armorcore node Kinc/make -g opengl --compiler clang --compile cd Deployment strip Krom ./Krom ../../build/krom

I can see that the build phase does not have any new line between commands… I’m thinking it could explain the build command errors… However, the derivation does have \n in the command:

$ nix show-derivation $(nix-instantiate)
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
{
  "/nix/store/6gvb0ysbk1ljwsqxxrrlgvrwbdl13v0w-armorpaint-20.01.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/4bhd8w8xphdihfd085irmiihqj1yyh8q-armorpaint-20.01"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/59chvvd768pkzc6diqkj5qnksmypd84h-bash-4.4-p23.drv": [
        "out"
      ],
      "/nix/store/iv5463szak8kk54fhlkk5bymx9zij3pp-source.drv": [
        "out"
      ],
      "/nix/store/r4fxw77hma70agwfvxvgrmsslamybbx3-nodejs-14.15.4.drv": [
        "out"
      ],
      "/nix/store/vp0crb9bxpqy8pb2hzv7i8ck6y6p41kq-stdenv-linux.drv": [
        "out"
      ]
    },
    "platform": "x86_64-linux",
    "builder": "/nix/store/cwnwyy82wrq53820z6yg7869z8dl5s7g-bash-4.4-p23/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "buildInputs": "/nix/store/nm15xz8bigq33d3x8r8lfrdlnvspmjrm-nodejs-14.15.4",
      "buildPhase": "node armorcore/make -g opengl\ncd armorcore\nnode Kinc/make -g opengl --compiler clang --compile\ncd Deployment\nstrip Krom\n./Krom ../../build/krom\n",
      "builder": "/nix/store/cwnwyy82wrq53820z6yg7869z8dl5s7g-bash-4.4-p23/bin/bash",
      "configureFlags": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "",
      "doInstallCheck": "",
      "name": "armorpaint-20.01",
      "nativeBuildInputs": "",
      "out": "/nix/store/4bhd8w8xphdihfd085irmiihqj1yyh8q-armorpaint-20.01",
      "outputs": "out",
      "patches": "",
      "pname": "armorpaint",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "src": "/nix/store/68v9kl0824hvwzjl9l8azmdsjjqf6khp-source",
      "stdenv": "/nix/store/hs7dqjfas58ly1hwnn014rfm33c9rk20-stdenv-linux",
      "strictDeps": "",
      "system": "x86_64-linux",
      "version": "20.01"
    }
  }
}

Any idea what’s happening?

Hmm, perhaps you have to move to an empty directory before starting the build

1 Like

Ohhhh this is very true, thanks a lot! It seems that this error raises if the folder already exist (so it occurs when you compile twice your program without cleaning the folder). Also, I solve the issue for \n, it is important to use "..." and to use eval "...":

If I just don’t put "...", I do not have any new line:

$ echo $buildPhase
node armorcore/make -g opengl cd armorcore node Kinc/make -g opengl --compiler clang --compile cd Deployment strip Krom ./Krom ../../build/krom

but if I had them, it works:

$ echo "$buildPhase"
node armorcore/make -g opengl
cd armorcore
node Kinc/make -g opengl --compiler clang --compile
cd Deployment
strip Krom

Similarly, if I just run that, it tries to run the command in one go:

$ $buildPhase
+ node armorcore/make -g opengl cd armorcore node Kinc/make -g opengl --compiler clang --compile cd Deployment strip Krom ./Krom ../../build/krom

if you just run "$buildPhase", it will try to find an executable whose name is this whole line (not possible), so you really want to call:

eval "$buildPhase"

Problem solved, thanks!

PS: for people interested by a derivation to build armorpaint, here is it:

{ lib,
  clangStdenv,
  fetchFromGitHub,
  # Tool to compile
  nodejs,
  autoPatchelfHook,
  git, # required to avoid errors
  pkg-config,
  # Libraries
  gtk3,
  pango,
  cairo,
  gdk-pixbuf,
  atk,
  glib,
  harfbuzz,
  alsaLib,
  # Wrapper
  runtimeShell,
  makeDesktopItem,
  symlinkJoin,
  wrapGAppsHook # Deal with gtk-3 app (otherwise, missing icons and crash when saving)
}:

let
  armorPaint = clangStdenv.mkDerivation rec {
    pname = "armorpaint";
    version = "20.01";

    src = fetchFromGitHub {
      owner = "armory3d";
      repo = "armorpaint";
      rev = "95453348b2";
      sha256 = "sha256-eaQhkD/IFjZDcSR8QMg4BdtsGDRjwIJAZUJt9uGEmB8=";
      fetchSubmodules = true;
      leaveDotGit = true; # Git is used at build time.
    #   deepClone = true; # Use if you want to clone the full history (depth > 1)
    };

    nativeBuildInputs = [
      autoPatchelfHook
      pkg-config
      git # Required at compile time, to avoid "Uncaught exception Eof" error
      wrapGAppsHook
    ];
    buildInputs = [
      nodejs
      gtk3
      gdk-pixbuf
      pango
      cairo
      atk
      glib
      harfbuzz
      alsaLib
    ];

    patchPhase = ''
      # The file armorcore/Kinc/Tools/kincmake/node_modules/physical-cpu-count/index.js
      # runs lscpu to get the number of cores to use to compile. This is not possible in Nix
      # (breaks determinism), and one must use instead the NIX_BUILD_CORES variable.
      cat > armorcore/Kinc/Tools/kincmake/node_modules/physical-cpu-count/index.js << EOF
      'use strict'
      let amount = $NIX_BUILD_CORES
      module.exports = amount
      EOF
    '';

    # gcc/clang will automatically consider these new flags (thanks clever!) without even modifying the Makefile.
    # That way, we don't need to patch the project.addIncludeDir("/usr/*"); lines in armorcore/kincfile.js
    # Some of the library are automatically added in $NIX_CFLAGS_COMPILE... but it's not always the case,
    # So we need to use that trick: The reasons is that nix automatically puts stuff in the $out/include folder
    # but some programs like gtk put the code inside a subfolder (for instance to deal with gtk2 vs gtk3).
    buildPhase = ''
      NIX_CFLAGS_COMPILE="$(pkg-config --cflags gtk+-3.0 glib-2.0 pango cairo) $NIX_CFLAGS_COMPILE"
      echo ">>> I'm patching elf files."
      autoPatchelf .
      echo ">>> I'm running armorcore/make. Note: this seems to fails if not in the actual git repo."
      node armorcore/make -g opengl
      echo ">>> I'm going in armorcore folder"
      cd armorcore
      echo ">>> I'm running Kinc/make"
      node Kinc/make -g opengl --compiler clang --compile
    '';

    installPhase = ''
      mkdir -p $out/{bin,opt/Deployment,opt/build,share/icons/hicolor/256x256/apps}
      cd ..
      cp -r armorcore/Deployment/* $out/opt/Deployment/
      cp -r build/* $out/opt/build
      # Create the wrapper that starts armorpaint
      cat > $out/bin/armorpaint <<EOF
      #!${runtimeShell} -e
      $out/opt/Deployment/Krom $out/opt/build/krom/ "$@"
      EOF
      chmod +x $out/bin/armorpaint
      # Copy the icon for the .desktop file
      cp armorcore/icon.png $out/share/icons/hicolor/256x256/apps/armorpaint.png
    '';
    };
  armorPaintDesktop = makeDesktopItem {
    name = "armorpaint";
    desktopName = "ArmorPaint";
    exec = "${armorPaint}/bin/armorpaint";
    icon = "armorpaint";
  };
in symlinkJoin {
  name = "armorpaint";
  paths = [ armorPaint armorPaintDesktop ];
}
1 Like