My subflake does not build properly? or How to package monorepo submodules using Nix?

Hello all!

I am trying to nixify a modular monorepo using multiple flakes.
This seems like the ideal approach because it enables nix run to work with each sub-directory, among other things. Read Nix sub-flakes design - HackMD to see the reasons this design is important to support.

The directory structure looks like:

root
└ .git
└ package1
  └ derivation.nix
  └ flake.nix
  └ ...
└ package2
  └ derivation.nix
  └ flake.nix
  └ ...
└ flake.nix
└ ...

I found a good GitHub issue thread on this subject, with working syntax in a comment by @aviallon at Allow flakes to refer to other flakes by relative path · Issue #3978 · NixOS/nix · GitHub
So I add child flakes to the inputs of the parent flake like:

  <...>
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
    subflake = {
      url = "git+file:.?dir=subflake";
      inputs.nixpkgs.follows="nixpkgs";
      inputs.flake-utils.follows="flake-utils";
    };
  };
  <...>

This works, for the most part.
I can use pkgs.mkShell { inputsFrom } to combine a child’s shell into the parent shell.

Parent flake.nix:

    <...>
    devShells.default = pkgs.mkShell {
      packages = [
        pkgs.podman
        pkgs.gzip
        pkgs.skopeo
      ];
      inputsFrom = [
        subflake.devShells.${system}.default
     ];
    };
    <...>

So nix develop works.

However, the child flakes seem to not resolve nixpkgs correctly outside of this use case.
Both pkgs.callPackage and pkgs.writeShellApplication fail to resolve.

Steps To Reproduce

Copy the following code into a file, ./mk-subflake-test.sh:

#!/bin/sh

# make the project directories
mkdir -p subflake-test/subflake

# enter the project directory
cd subflake-test

# track the repository by git
git init

# create the parent flake
cat > flake.nix << EOF
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
    subflake = {
      url = "git+file:.?dir=subflake";
      inputs.nixpkgs.follows="nixpkgs";
      inputs.flake-utils.follows="flake-utils";
    };
  };
  outputs = { self, nixpkgs, flake-utils, subflake, ... }:
  flake-utils.lib.eachDefaultSystem (
    system: let
      pkgs = import nixpkgs { inherit system; };
    in rec {
      packages.default = pkgs.writeShellApplciation {
        name = "subflake-test";
        runtimeInputs = [ subflake ];
        text = ''
          \${pkgs.lib.getExe subflake.packages.\${system}.default}
        '';
      };
      apps.default = flake-utils.lib.mkApp { drv = packages.default; };
    }
  );
}
EOF

# create the child flake (subflake)
cat > subflake/flake.nix << EOF
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils, ... }:
  flake-utils.lib.eachDefaultSystem (
    system: let
      pkgs = import nixpkgs { inherit system; };
    in rec {
      packages.default = pkgs.callPackage ./test.nix {};
      apps.default = flake-utils.lib.mkApp { drv = packages.default; };
    }
  );
}
EOF

# create the nix derivation
cat > subflake/test.nix << EOF
{ pkgs }: pkgs.writeShellApplication {
  name = "subflake-test";
  text = ''
    #/bin/sh
    echo "Hello, World!"
  '';
}
EOF

# track the project files using git
git add .

Then execute the file using:

chmod +x ./mk-subflake-test.sh && ./mk-subflake-test.sh

Now enter the project directory using cd subflake-test and then use nix run;
Observe the error returned.

Now, nix run in subflake-test returns:

 > 16:52 subflake-test git:(main) X nix run
warning: Git tree '/home/admin/code/subflake-test' is dirty
warning: Git tree '.' is dirty
warning: creating lock file '/home/admin/code/subflake-test/flake.lock'
warning: Git tree '/home/admin/code/subflake-test' is dirty
error:
       … while evaluating the attribute 'default'

         at /nix/store/hzwacgdhqd752zyg2hyv9rrk1b28hb48-source/flake.nix:16:7:

           15|     in rec {
           16|       packages.default = pkgs.writeShellApplciation {
             |       ^
           17|         name = "subflake-test";

       error: attribute 'writeShellApplciation' missing

       at /nix/store/hzwacgdhqd752zyg2hyv9rrk1b28hb48-source/flake.nix:16:26:

           15|     in rec {
           16|       packages.default = pkgs.writeShellApplciation {
             |                          ^
           17|         name = "subflake-test";
       Did you mean writeShellApplication?
 > 16:52 subflake-test git:(main) X

Yet nix run in subflake-test/subflake returns:

 > 16:52 subflake-test/subflake git:(main) X nix run
warning: Git tree '/home/admin/code/subflake-test' is dirty
warning: creating lock file '/home/admin/code/subflake-test/subflake/flake.lock'
warning: Git tree '/home/admin/code/subflake-test' is dirty
Hello, World!
 > 16:52 subflake-test/subflake git:(main) X 

Why is this not working?
How should I be using sub-flakes instead?
If it’s impossible to use sub-flakes, how should I organize my code with multiple submodules?

Thank you for your time.

Solution

There 3 options.

  1. Call nix flake lock --update-input subflake for each subflake whenever it’s modified (source: from a GitHub Issue).
  2. Store each package inside their own separate git repositories.
  3. Don’t use sub-flakes.

I decided to stop trying to use sub-flakes. I outlined the approach I used instead in this post.

1 Like

It tells you the typo :slight_smile:

3 Likes

Wow.
Well, it would seem you are correct.
In fact, my example works after fixing the typo…
It’s very funny that I missed that; I even included the message in my post.

I still have a similar issue in my actual project, so I will have to go figure out my issue, or create a minimum working example of the actual error I have.

In the real project, I get:

 > 22:02 lacuna git:(main) nix run
error:
       … while calling the 'getAttr' builtin

         at /builtin/derivation.nix:19:19: (source not available)

       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: attribute 'callPackage' missing

       at /nix/store/r0mbz5w1rg4vm4f4ws7n4x11fmca2zfc-source/sveltekit/flake.nix:17:15:

           16|       packages = {
           17|         dev = nixpkgs.callPackage ./nix/dev.nix { inherit name version; };
             |               ^
           18|         server = nixpkgs.callPackage ./nix/server.nix { inherit name version; };
 > 22:03 lacuna git:(main)

I’ll report back.

Afaik this bug remains unresolved, so actually working with subflakes is going to remain a royal pain. I strongly recommend against it, for now.

3 Likes

Well depends what nixpkgs means here, if it’s referring to the arg of the flake outputs function corresponding to the nixpkgs input, then you have to write nixpkgs.legacyPackages.x86_64-linux.callPackage (replace the system attribute accordingly) not nixpkgs.callPackage. If it’s something else then we have to see the code.

Though yes, I agree with @tejing that subflakes currently appear only half-implemented, including longstanding issues around locking.

Thank you for the note. As I understand, this implementation detail does not affect my use case.

It is okay if the child flakes have different lockfiles from the parent flake, as long as their build result in the Nix store is generated with different hashes (it should be, since dependencies are dfiferent).

It’s impossible for it not to affect you if your parent flake refers to your subflake (or really any reference between flakes in the same repository). It’s not that the child has a “different lockfile” from the parent. It’s that the parent locks a version of the child from a different commit of the repo, which nix also doesn’t even know how to find because the only reference it has is a local path within the flake. So you can very easily end up in a state where it’s actually not possible for nix to build the flake’s outputs without happening to have the right commit already cached in the nix store from previous work (which you probably do… until a gc runs and suddenly you don’t…). As a practical matter, it makes working with subflakes infuriatingly difficult and error-prone.

2 Likes

@tejing You are correct. Thank you!

My issue was exactly what you described. The first build of the project had a syntax error. I fixed it, but the project continued to report the error, because the subflake didn’t ever get re-built.

Deleting the root flake.lock and then using nix run fixed my build issue, cause it forced the subflake reference to update. Since the subflake dependency doesn’t automatically re-build within a parent flake, the flake is cached once on the first run, and then used forever OR could fail to resolve as you described.

This leaves me with options:

  1. Call nix flake lock --update-input subflake for each subflake whenever it’s modified (source: from a GitHub Issue).
  2. Store each package inside their own separate git repositories.
  3. Don’t use sub-flakes.

Option 1 is not a very good pattern, unless I were using Nix inside another scripting tool like Bash.

Option 2 is not possible for my use case.

So I must re-organize my project without subflakes. My question remains then,

If it’s impossible to use sub-flakes, how should I organize my code with multiple submodules?

So it turns out that Nix doesn’t have clear project organization recommendations. Clear truths are that nix-build targets ./default.nix in a given directory and nix-shell targets ./shell.nix. The structure that nixpkgs provides is what most people base their configurations off of, but Nix itself is not largely opinionated.

So I figured out how to avoid subflakes by designing my own project structure.
I now know how to use a single flake.nix to package multiple submodules.
Hopefully this post will help others design their solution quicker.

Here is a link to the open source project.

Nix Flake Scripts

The following commands can be used in a shell with Nix installed and Flakes enabled.
These commands work from any subdirectory in the repository too.
Because Nix manages all dependencies, it is the only tool required to be installed manually.

Command Description
nix run alias for nix run .#start -- dev
nix run .#start start the app locally
nix run .#deploy deploy the app
nix run .#init initialize the app for deployment
nix develop start a shell with access to all dependencies
nix develop .#submodule start a shell with access to only dependencies for a specific submodule

Scripts are declared in flake.nix:

{
  description = "Project.";
  inputs = {
    # get new revision # using:           git ls-remote https://github.com/<repo_path> | grep HEAD
    # pin dependency to revision # like:  url = "github:numtide/flake-utils?rev=#"
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils, ... }: let
    name = "project";
    version = "0.0.0";
    utils = flake-utils;
  in utils.lib.eachDefaultSystem (
    system: let
      pkgs = import nixpkgs { inherit system; };
      submodule = import ./submodule { inherit pkgs; };
      # anotherSubmodule = import ./anotherSubmodule { inherit pkgs; };
    in rec {
      packages = {
        # help
        # init
        # deploy
        start = pkgs.callPackage ./scripts/start.nix {
          inherit name version;
          webServer = submodule;
          # database = anotherSubmodule;
        };
        default = packages.start;
      };
      apps = {
        # help
        # init
        # deploy
        start = utils.lib.mkApp { drv = packages.start; };
        default = utils.lib.mkApp { drv = packages.start.override { cliArgs = [ "dev" ]; }; };
      };
      devShells = {
        submodule = submodule.devShells.default;
        # anotherSubmodule = anotherSubmodule.devShells.default;
        root = pkgs.mkShell {
          packages = [
            # examples:
            # pkgs.podman
            # pkgs.gzip
            # pkgs.skopeo
          ];
        };
        default = pkgs.mkShell {
          inputsFrom = [
            devShells.root
            devShells.submodule
            # devShells.anotherSubmodule
          ];
        };
      };
    }
  );
}

Each script is defined in scripts/. For example, scripts/start.nix:

{
  pkgs,
  name,
  version,
  webServer,
  # database,
  cliArgs ? []
}: pkgs.writeShellApplication {
  name = "${name}-start-${version}";
  runtimeInputs = [
    webServer
  ];
  text = ''
    set -- "$@" ${pkgs.lib.strings.concatStringsSep " " cliArgs}
    ${pkgs.lib.getExe webServer} "$@"
  '';
}

Then each submodule can declare its own scripts in submodule/default.nix:

{
  pkgs ? import <nixpkgs> {}
}: let
  # I pull in name and version numbers from a Node.js `package.json`, but you could set those manually instead
  nodePackage = builtins.fromJSON (builtins.readFile ./package.json);
  name = nodePackage.name;
  version = nodePackage.version;
in rec {
  packages = {
    start = pkgs.callPackage ./scripts/start.nix {
      inherit pkgs name version;
    };
    default = packages.start;
  };
  devShells.default = import ./shell.nix { inherit pkgs; };
}

To be compatible with nix-shell, we also provide submodule/shell.nix:

{
  pkgs ? import <nixpkgs> {}
}: pkgs.mkShell {
  packages = [
    # examples:
    # pkgs.pnpm
    # pkgs.nodejs
    # pkgs.nix-prefetch-docker
  ];
}

And individual scripts are defined in submodule/scripts/. For example:

{
  pkgs,
  name,
  version,
}: let
  start = {
    main = pkgs.writeShellApplication {
      name = "${name}-start-main-${version}";
      runtimeInputs = [];
      text = ''
        echoerror() {
          echo "Error in start-main:" "$@" 1>&2;
        }
        interpret_args() {
          while [[ $# -gt 0 ]]; do
            case $1 in
              *)
                additional_args+=("$1")
                shift
              ;;
            esac
          done
          set -- "''${additional_args[@]}"
        }
        main() {
          interpret_args "$@"
          if [[ -n "''${script:-}" ]]; then
            "$script" "''${additional_args[@]}"
          else
            echoerror "No valid script identified among arguments:" "''${additional_args[@]}"
            exit 1
          fi
        }
        main "$@"
      '';
    };
    # TODO list other scripts here that main can call, then install them in installPhase
  };
in pkgs.stdenv.mkDerivation rec {
  pname = "${name}-start";
  inherit version;
  src = ./.;
  phases = [ "installPhase" ];
  buildInputs = [ start.main ];
  installPhase = ''
    mkdir -p $out/bin/
    cp ${pkgs.lib.getExe start.main} $out/bin/main.sh
    chmod +x $out/bin/main.sh
  '';
  meta.mainProgram = "main.sh";
}

Because sub-modules take arguments into a bash script from the parent module’s bash script, you can perform any branching logic in here. Modules are hot-swappable too, as long as you make a small compatibility layer using Nix as I’ve described.

Finally, I can use Nix to package and develop in my monorepo.

Why do this?

I need to use Nix (dockerTools.streamLayeredImage) to package my project into Docker images. It seems convenient to use Nix to deploy my project to the Docker containers too, and so packaging the entire project using Nix just makes sense.

Using Nix as the package manager means the software Just Works™ on everyone’s machine once they install Nix and enable Flakes.

Nix is highly configurable. Capable of version pinning, patching, and sub-scripting at every step, it is the most script-able package manager I’ve ever used.