Get Nix Flake to include git submodule

Help
1

I have a directory that looks like this:

.git/
submodule/
| .git/
| ...
stuff/
flake.nix
...

I want the submodule (which is not a flake) to be included in the flake source directory. What’s the best way to do this? I’ve tried the various permutations of url options from “nix flakes: add support for git submodules”.

1 Like
2

Use submodules=1 in the query.

If this does not work, please share a minimal reproducible example that we can do some testing with.

1 Like
3

What’s the input url supposed to be for a submodule in the flake’s directory? Checking out from remote works (very inefficient), and this specific combination of:

url = "file:///[snip]/submodule?submodules=1";
type = "git";

works, but these failed:

inputs.submodule = {
    flake = false;
    url = "git+file:///[snip]/submodule?submodules=1";
}; # Fails w/ `error: opening file '[snip]/submodule/.git/config': Not a directory`

inputs.submodule = {
    flake = false;
    url = "git+path:submodule?submodules=1";
}; # Unsupported

inputs.submodule = {
    flake = false;
    url = "path:submodule?submodules=1";
    type = "git";
}; # Unsupported

I’d rather not have to specify the absolute path to the submodule.

4

You do not add the submodule itself as an input, you need to add the submodules=1 query parameter to the input that has the submodule.

If this is your “self”, you need to provide the query parameter on each CLI invocation.

nix build '.?submodules=1'
2 Likes
5

That worked! And is… very unintuitive.

EDIT: Is there any way to avoid manually specifying that every time?

1 Like
6

The only way I am aware of that would remove the necessity of specifying submodules=1 is to remove the submodule.

And I do not consider this unintuitive. You have to specify the argument to the input that has the submodules, if this input is self, then the only place where you can specify that is the CLI.

7

I’m definitely a nix novice, so you probably know more than me. But, I don’t think I’ve ever passed ?something to the nix CLI before and it’s undocumented AFAIK (excluding the github issue where people are similarly confused).

8

I agree that it is not convinient. Still, it is not unintuitive, as this is the only place where you specify selfs URL.

The CLI is also where you are able to change its schema.

Though my stance remains: the best way to deal with git submodules is to get rid of them…

9

I guess “intuitive” is in the eye of the beholder, because the only place I specify self’s url is… nowhere. I don’t specify it. I expect nix develop (and direnv’s use flake) to just work. :slight_smile:

10

There you implicitely specify .. If you want to use submodules, you have to be explicit about that.

11

Ok, I understand, but I do consider this to mean that flakes just don’t properly support git submodules and will work around it to the best of my ability.

3 Likes
12

I’m running into this also.

In a real-world context, requiring users of my flake to know that they must add a “special” command-line parameter ?submodules=1 creates a condition where things “don’t work” by default.

I imagine it should be possible to set submodules=true within the flake, but my attempt so far yields an error:

  outputs = {
    self
  } @ inputs: let
    # error: attribute 'override' missing
    self = inputs.self.override (old: { submodules = true; }); # overrideAttrs also doesn't work
  in
    {
      # outputs
    }

Another approach might be to specify the local directory as an input, and give it a submodules=1 parameter there.

This input specification works, but without including the submodule:

  inputs = {
    src = {
      url = "file:.?submodules=1";
      type = "git";
      flake = false;  # not including this results in inifite recursion
    };
  };

Another approach (and this may be the cleanest?) would be to set submodules as a toplevel flake variable, eg:

{
  submodules = true;

  inputs = {};
}

Interestingly, a nix repl already sees the submoduels toplevel attribute:

╰─➤  nix repl                                                                                                                 1 ↵
Welcome to Nix 2.18.1. Type :? for help.

nix-repl> :lf .
Added 14 variables.

nix-repl> submodules
false

nix-repl> inputs
{ flake-utils = { ... }; nixpkgs = { ... }; }

nix-repl>
4 Likes
13

Looks like I’m not the only one seeing this.

The upshot is that as of Nix 2.20 the need for ?submodules=1 will go away entirely: feat: support managing submodules via inputs.self by tomberek · Pull Request #7862 · NixOS/nix · GitHub

1 Like
14

I think the fact that there’s no way to make self include submodules without having to inform the developer/user to specify ?submodules=1 is both unintuitive and inconvenient.

A normal behavior would be to call:

nix build github:example-org/example-pkg

And receive a built example-pkg binary. But instead you get errors about missing submodules.

Yes, it is unintuitive and this should be configurable in the flake.nix file.

I see someone has made an attempt to support this, but it’s not really moving:

15

I’m wondering how can I use a flake as an input for my system configuration, where such a flake needs “?submodules=1” to build?

1 Like
16

This worked for me, but haven’t tested with a remote eg. github:

url = "git+file:///PATH?submodules=1";
2 Likes
17

I’ve realized that one way to avoid the .?submodules=1# UX nightmare is to declare the submodule as a discrete file-based input:

inputs = {
  # ...
  subproject = {
    url = "git+file:subproject"; # the submodule is in the ./subproject dir
    flake = false;
  };
}

Then, any derivation(s) which rely on the submodule need to be extended to take the submodule as an input and need to deal with it manually; eg:

  buildPhase = let
    prefix = python3.sitePackages; # eg: lib/python3.11/site-packages
  in ''
    # copy entire package as a library
    mkdir -p $out/${prefix}
    ${rsync}/bin/rsync -rl ${src}/ $out/${prefix}/
    # same for 'subproject' submodule
    mkdir -p $out/${prefix}/subproject
    ${rsync}/bin/rsync -rl ${subproject}/ $out/${prefix}/subproject/

    # rest of build phase

This will work locally and in CI, as long as the submodule is checked out.

Then, when passing this toplevel project flake as an input to another flake, use submodules=1, eg:

      url = "github:owner/project_name?ref=branch/name/with/slashes&submodules=1";

      # this alternative also works:
      url = "git+ssh://git@github.com/owner/project_name.git?ref=branch/name/with/slashes&submodules=1";
6 Likes
18

Hey, dont wanna hijack your thread but do I need to build the subproject after referencing as input?
Using the suggestions from your last answer:

inputs = {
  # ...
  subproject = {
    url = "git+file:subproject"; # the submodule is in the ./subproject dir
    flake = false;
  };
}

but all I get is a error message with “/subproject: No such file or directory”. Already update, synced and reinit the submodule.

19

hmm, this really should work as long as the submodule:

  • is present in .gitmodules (git submodule add URL)
  • the commit adding the submodule to .gitmodules and pinning its version is present in the superproject’s git history (git add ./submodule && git commit)
  • the submodule is checked out (files present on disk: git submodule update --init)

Let me know if that solves it

20

Thanks for your response. Tried everything you have listed and a lot more but did not got it working unfortunately. Fixed it with a workaround fetching the git repo now from remote instead using a submodules, works flawlessly for my purpose.

1 Like