Breaking up a monolithic flake

Help
1

I am working on a project (let’s call it WRAPPER) which consists entirely of APIs and utilites which help with writing code in a specific scientific simulation framework (let’s call it FRAMEWORK). As such my code is useless without the framework which it enhances.

In case it’s relevant, this is all written in C++, and the framework uses cmake.

For my initial convenience, I started off with a single flake.nix which installs the simulation framework, the test framework I’m using, some development tools, etc. This has served me well in all manner of use cases during the initial phase of development, during which WRAPPER, the FRAMEWORK application which used it, and the client code which used the latter, all lived in the same repository.

Now that I need to distribute WRAPPER reliably and conveniently for others to use in other contexts, I am wondering how to split my monolithic flake into smaller components which cater for different aspects of WRAPPER’s use, to get the ability to provide

  1. WRAPPER itself
  2. a development environment for working on WRAPPER
  3. everything necessary for testing WRAPPER on CI
  4. a development environment for projects that use WRAPPER to write applications in FRAMEWORK
  5. everything needed by users of projects written with WRAPPER+FRAMEWORK, i.e. something that projects in category 4 would provide to their users.
  6. a template for starting projects in category 4

I imagine the following differences between the above

  • 3 vs 2

    • 3 is probably 2 minus LSP servers, debuggers, profilers etc.
    • 2 should use the in-tree WRAPPER sources directly; should 3 get WRAPPER via 1?

  • 4 vs 2

    • 2 should use the in-tree WRAPPER sources directly; 4 should get WRAPPER via 1
    • 4 may need extra dependencies which are irrelevant to 2

  • 5

    • may need extra dependencies which are irrelevant to the others
    • can do without the testing framework which is needed by 2,3 and 4

I strongly suspect that the baggage that I bring from having used the monolithic flake for a while, combined with my patchy grasp of Nix, is preventing me from seeing decent solutions.

I’ll also need this to work on Linux, MacOS with Apple Silicon and WSL2, but that’s probably a second-order issue at this point.

Can you offer any advice on how to organize this?

2

In this case would make sense change the order:
3 to be 2 and, 3 to be plus LSP servers, debuggers, profilers ?

Did you had time to look at flakes outputs schema?

{ self, ... }@inputs:
{
  # Executed by `nix flake check`
  checks."<system>"."<name>" = derivation;
  # Executed by `nix build .`
  packages."<system>".default = derivation;
  # Executed by `nix run . -- <args?>`
  apps."<system>".default = { type = "app"; program = "..."; };
  # Used by `nix develop .#<name>`
  devShells."<system>"."<name>" = derivation;
  # Used by `nix flake init -t <flake>`;
  templates.default = { path = "<store-path>"; description = ""; };
}

You can use different kinds of outputs for 1,2,3,…
And you can use self, to mix 1, 2 (like lib.mkShellself { packages = [ self.packages.default ]; })

And there are some nice ideas out there like

3

I wasn’t implying any order, merely stating the relationship between the two, in the sense that a=b+c means exactly the same as b=a-c, without implying anything about whether a should be made from b or the other way around. Algebra rather than assignment, if you like.

Yes, but I’m not sure that it is very relevant here. It describes the different roles played by various outputs in a single flake. I think (though it’s perfectly possible that I’m missing something) that I need a number of different flakes for different uses. For example 2, 4 and 5 in my list above all need devShells.<system>.default but each one needs it to contain something different.

I guess there’s a typo here? I’m not managing to divine exactly what you’re trying to say here.

Thanks. There’s a lot to digest there. Probably more than I can manage in the short term.

4

Yes, but in nix b + c is possible and easy

let a = mkDerivation {
  buildInputs = [ b c ];
  # ...
}; in a

While a - c also possible but harder.

let b = a.override (self: super: { 
  buildInputs = builtins.filter (i: i != c ) super.buildInputs
}); in b

Algebraically thinking, this b is not initial b but a’ (a = f(b, c) and a’ = f(b, not(c))). Anyway, a’ attend to your requirement of a - c package/environment but requires to download/build c, that you don’t want.

Your question is about breaking a monolithic flake, but it could be about a monolithic package. Since a single flake could have all your scenarios.

  1. packages.<system>.default (the wrapper)
  2. devShells.<system>.Dev (dev env for wrapper)
  3. checks.<system>.Test (test package for wrapper)
  4. devShells.<system>.default (dev env for your users, but could be just 6)
  5. packages.<system>.FramworkAndWrapper (I am not sure about this)
  6. templates.default (template to use 4, or template with 4)
  7. packages.<system>.Framework (the framework)

I’m not managing to divine exactly what you’re trying to say here.

Let me try the long version, you can use self, to mix 1, 2

{
  outputs: {self, nixpkgs, ...}: {
    packages.x86_64-linux.default = nixpkgs.legacyPackages.x86_64-linux.stdEnv.mkDerivation {
      # wrapper package
    };
    packages.x86_64-linux.framework = nixpkgs.legacyPackages.x86_64-linux.stdEnv.mkDerivation {
      # wrapper package
    };
    devShells.x86_64-linux.Dev = nixpkgs.lib.mkShell { 
      packages = [
        self.packages.x86_64-linux.default
        self.packages.x86_64-linux.framework
        nixpkgs.legacyPackages.x86_64-linux.LINTER
        # ...
      ];
    };
  };
}

But you can have different flakes as well

{
  inputs.wrapper.url = "path:/wrapper/";
  outputs: {self, wrapper, nixpkgs, ...}: {
    devShells.x86_64-linux.Dev = nixpkgs.lib.mkShell { 
      packages = [
        wrapper.packages.x86_64-linux.default
        wrapper.packages.x86_64-linux.framework
        nixpkgs.legacyPackages.x86_64-linux.LINTER
        # ...
      ];
    };
  };
}
1 Like