In-repo Git submodule pull while building derivation

Help
1

I am trying to write a derivation within a Git repository. The repository has submodules to pull before building and it seems that Nix is not copying .git in the build sandbox.

Here (1) they propose to use pkgs.fetchgit. I tried but Nix still cannot find .git when building and therefore cannot pull the submodules. Any input?

{ clangStdenv, gcc, fetchgit }:

clangStdenv.mkDerivation {
  name = "harden";
  version = "1.0.0";

  src = fetchgit {
    url = ./.;
    sha256 = "sha256-87XMZ7TWU4cXif+TiDel03AOjzCAIqbogpLwl+RDmZg=";
    fetchSubmodules = true;
  };
  nativeBuildInputs = [ gcc ];

  configurePhase = ''
    make -C BaseTools
  '';
}

svzgms4sxwir8lra267rmc1i1pnamgbs-source> exporting /nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source (rev HEAD) into /nix/store/00qyg4zyw5c10ah7rnn4pwilliw077yw-svzgms4sxwir8lra267rmc1i1pnamgbs-source
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Initialized empty Git repository in /nix/store/00qyg4zyw5c10ah7rnn4pwilliw077yw-svzgms4sxwir8lra267rmc1i1pnamgbs-source/.git/
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: '/nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source' does not appear to be a git repository
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: Could not read from remote repository.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Please make sure you have the correct access rights
svzgms4sxwir8lra267rmc1i1pnamgbs-source> and the repository exists.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: '/nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source' does not appear to be a git repository
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: Could not read from remote repository.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Please make sure you have the correct access rights
svzgms4sxwir8lra267rmc1i1pnamgbs-source> and the repository exists.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: '/nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source' does not appear to be a git repository
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: Could not read from remote repository.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Please make sure you have the correct access rights
svzgms4sxwir8lra267rmc1i1pnamgbs-source> and the repository exists.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: '/nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source' does not appear to be a git repository
svzgms4sxwir8lra267rmc1i1pnamgbs-source> fatal: Could not read from remote repository.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Please make sure you have the correct access rights
svzgms4sxwir8lra267rmc1i1pnamgbs-source> and the repository exists.
svzgms4sxwir8lra267rmc1i1pnamgbs-source> Unable to checkout HEAD from /nix/store/mb71324hcd6cj6mff9igickw7vy9ah40-svzgms4sxwir8lra267rmc1i1pnamgbs-source.
2

I can see from the output that this is a flake, so you need to make nix fetch the submodules when copying the flake. By the time it’s even looking at your derivation’s code, .git is already gone, and you can’t really change that without breaking reproducibility. You can add a parameter to the flake url (?submodules=1 iirc) to force nix to fetch submodules, but that can get tedious very fast when working on the cli. Honestly, the best path is usually to just not use submodules, if at all possible. Flake inputs fill much the same role in the end.

1 Like
3

Thanks for the input @tejing.

The derivation is not built from a flake input. The flake is just a skeleton for the derivation:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
        };

        shellWithPkgs = packages: pkgs.mkShell {
          inherit packages;
        };
      in
      {
        devShells.default = shellWithPkgs [ pkgs.hello ];
        defaultPackage = pkgs.callPackage ./harden.nix {};
      });
}

Unfortunately, I cannot work without the submodules as I do not control the code base.

4

Then I think your only option is to specify .?submodules=1 as a flake url on all your nix commands that interact with the flake, tiresome as that may get.

1 Like
5

I tried with `nix build “.?submodules=1” but no luck yet.

$ nix build ".?submodules=1" -L                                                                                                                                                                                 
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> exporting /nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source (rev HEAD) into /nix/store/cim6j5v4g0fqh1fhkj2yk1mhmd69ky6y-jfqka9gmddbxm1i038alpc9vna0a8l0g-source
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Initialized empty Git repository in /nix/store/cim6j5v4g0fqh1fhkj2yk1mhmd69ky6y-jfqka9gmddbxm1i038alpc9vna0a8l0g-source/.git/
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: '/nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source' does not appear to be a git repository
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: Could not read from remote repository.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Please make sure you have the correct access rights
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> and the repository exists.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: '/nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source' does not appear to be a git repository
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: Could not read from remote repository.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Please make sure you have the correct access rights
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> and the repository exists.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: '/nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source' does not appear to be a git repository
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: Could not read from remote repository.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Please make sure you have the correct access rights
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> and the repository exists.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: '/nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source' does not appear to be a git repository
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> fatal: Could not read from remote repository.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Please make sure you have the correct access rights
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> and the repository exists.
jfqka9gmddbxm1i038alpc9vna0a8l0g-source> Unable to checkout HEAD from /nix/store/f313x715w3bdy9pl7s314pyzp3zi7fdj-jfqka9gmddbxm1i038alpc9vna0a8l0g-source.

I also tried to use nix-build with the compat flake template, same results:

$ nix-build                                                                                                                                                                                                     
these 2 derivations will be built:
  /nix/store/73xk1arh25rbd03zl88chm0pwywspg70-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source.drv
  /nix/store/dqardfzdygvclvw7ndc393q68alm7y51-harden.drv
building '/nix/store/73xk1arh25rbd03zl88chm0pwywspg70-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source.drv'...
exporting /nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source (rev HEAD) into /nix/store/6w12pi6f96ch1cw3n01zcf68bdf9jbsc-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source
Initialized empty Git repository in /nix/store/6w12pi6f96ch1cw3n01zcf68bdf9jbsc-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source/.git/
fatal: '/nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: '/nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: '/nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: '/nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Unable to checkout HEAD from /nix/store/qf3hvbqmx649sc55v0pxlg76gxghr7zd-nlzajcmvb9fli6jlzfgfcfhx4fyrll8h-source.
6

You also need to throw out the fetchGit call. That was never going to work (in a flake). Just set src = ./.;.

1 Like
7

Fantastic, it worked.

Thank you very much!