Omniflake: 12 thousand flakes

You can check it out at https://omniflake.com

Disclose AI use so I can avoid all AI centric comments :melting_face:

19 Likes

Now that you have every flake, and every package version for nixpkgs, it’s time to reach for the ultimate flake: Every version of every flake in one flake

4 Likes

I love how you are continuously pushing the boundaries of what we can do with Nix and are eager to experiment. Another intriguing idea. Thank you :heart: I played with this yesterday, and it seems to work nicely. I am wondering about two things, though:

  1. Deduplicating all dependency flakes automatically: What I am unsure of is how to configure automatic follows in inputs.omniflake.flakes for not just the few common flakes (nixpkgs, flake-utils, etc.), but everything that you already use (explicitly or implicitly through other input flakes). E.g., flake1 uses rust-overlay, and flake2 as well, and you explicitly do not use rust-overlay. I would like inputs.omniflake.flakes.flake1 and inputs.omniflake.flakes. flake2 to share the same rust-overlay input (the pin from the omniflake, presumably). I can do this manually with overrides, etc., but the automatic resolution when using inouts.omniflake.flakes would be great. Of course, this can bring other complications (compatibility issues between different versions of rust-overlay, etc.), but I think some people would find use cases for this.

  2. Naming scheme design choices: I am wondering about the naming scheme for the included flakes. Instinctively, I do not exactly like that the names are not fully qualified. Whichever flake of a given name comes first gets the simple <flake-repo-name>, and others afterwards get <flake-repo-name>-<owner-name>(-<source-forge-name>). Personally, I would prefer fully qualified names (up to the source force, maybe; ala Flatpak reverse domain-style naming, and Nix flake references scheme; I would even go as far as truly using reverse naming scheme), and possibly manual (not automated) configuration of predefined names for common and well-known projects such as nixpkgs, devenv, systems, flake-utils, etc… Introduction of new reserved names on a per-project basis if deemed common/interesting/important enough. Currently, the names are not reused, so there is no issue with the inputs changing under the user’s hands, but I still find it counterintuitive. Alternatively, in the approach above, one expects that the reserved names can be reused if, e.g., the old project is deprecated and a new fork takes over (with the intention of migrating users). Was there some special reason for the chosen approach of not using fully qualified names? E.g., github:xremap/nix-flake now reserves the name nix-flake, other projects are named nix-flake-<owner-name>, and the xremap name is never used in the index, making it undiscoverable.

Any thoughts on this?

2 Likes

I will try to get to most of your answers (may be multiple messages).

deduplication
Please see Unification Ā· omniflake and tell me what’s missing.

  1. omniflake.flakes.<name> substitutes five inputs into the flake, matched by input name, at every depth of its input graph:

    nixpkgs  flake-utils  systems  flake-parts  flake-compat
    

I find this the most ergonomic.

  1. omniflakes.pinned.flakes is the exact flake

  2. lib.withOverrides returns every flake under an override set of your choice; lib.load does the same for one flake.

    let
      mine = omniflake.lib.withOverrides {
        nixpkgs = nixpkgs;
        nixpkgs-stable = nixpkgs-stable;
        nixpkgs-unstable = nixpkgs;
      };
    in
    mine.sops-nix.nixosModules.sops
    
    

    You can also override per flake

    omniflake.lib.load "nh" { nixpkgs = nixpkgs-stable; }
    

The interface seems to be exactly what I would want, except I would expect omniflake.flakes to substitute everything (e.g., rust-overlay), not just the 5 main flakes. Basically integration of something like your nix-auto-follow. omniflake.lib.withOverrides is basically what I want, but now I want everything to be overriden automatically (make a closure of what is needed by everything included, and then resolve to only a single flake instance for everything, if possible. One can then manually override cases where the substitute all fails, for example. Finally, the substitution of the main few flakes is a great idea in and of itself. So, maybe I would imagine using both the the override everything and override only the 5 main flakes?

The problem with having everything unify is ā€œwhat commit should we pickā€ ?
nix-auto-follow works because the top-level flake decides the commit to pipe into everything else.

That’s why withOverrides does the same thing. Without it there’s no way to know what to pick…

Those 5 flakes nixpkgs, flake-utils etc.. are top-level inputs of omniflake and that’s why they are alread ā€œpickedā€ at a particular revision to pipe down.

Never mind I figured it out!

I was afraid you will say something like the one above. Fantastic to hear you found a solution. Theoretically, I think that some fix-point calculation should be applicable here (as a worst case; it might be much simpler).

Please see Unification Ā· omniflake

omniflake.unified.<name> substitutes on every name the index knows, not just the five. A graph reaches one home-manager, one disko, one treefmt-nix — the revision the pipeline pinned — instead of the revision each author happened to lock.

It’s a bit sharp (might not always work) so I don’t add it to the top level README

3 Likes

Yeah, that seems perfect. It covers the whole of question 1 from my original reply. Much love :heart: I will try it on my configs.

for search.nixos.org, this discoverability aspect of flakes had kind of been an outstanding challenge.

would you mind if we looked into what we could add there based on your index?

3 Likes

I just tried one and i get an error.

nix run ā€˜github:fzakaria/omniflake#flakes.fhs-environment.packages.x86_64-linux.default’

error:

flake ’

github:fzakaria/omniflake

’ does not provide attribute

ā€˜apps.x86_64-linux.flakes.fhs-environment.packages.x86_64-linux.default’, ā€˜packages.x86_64-linux.flakes.fhs-environment.packages.x86_64-linux.default’, ā€˜legacyPackages.x86_64-linux.flakes.fhs-environment.packages.x86_64-linux.default’ or ā€˜flakes.fhs-environment.packages.x86_64-linux.default’

Do I need to do something first? I dont think my environment is special.

That’s a very old flake (5 years) and default didn’t exist yet as a convention.

$ nix build 'github:fzakaria/omniflake#pinned.fhs-environment.packages.x86_64-linux.fhs-environment'

You can access the packages directly.

Sure; it just scrapes GitHub with the API based on stars.
I have some hard-cap since 1-10 stars there’s like 100k+ projects.

I also do some ā€œfilteringā€ (AI-vibed) to remove what we deem to be personal Nix configs.

2 Likes

I just reworked the API a bit and added full qualified naming now.

omniflake.flakes."github:NixOS/nixpkgs"

# also this way
omniflake.github.flakes.nix-community.home-manager

1 Like

Awesome :heart: Exactly what I envisioned. I am happy to report that I have successfully replaced quite a number of my inputs, with my lockfile going from just slightly shy of 100 locked nodes to 22, of which most are just pinned forwarders to omniflake’s index or not flakes at all; basically, only 2 projects currently fail to be indexed by omniflake for ambiguous reasons (github:helix-editor/helix and github:xremap/nix-flake, specifically). GitHub finally stopped rate-limiting me on each nix flake update due to the sheer number of flakes pulled from there, and all seems to be running well even after using omniflake.unified for everything. I am working on a PR to fix the issues I have encountered so far with the index, leaving only the two issues above unresolved for me. Some incorrect names, some missing flakes. A few issues have already been fixed by you and others. Overall, amazing :heart:

Intuitively, shouldn’t the github and flakes part be swapped? Is there a specific reason for the order to be like this?

Thanks for the contribution. Merged!

As for why github.flakes I think was just in case there was a popular flake called github …

Otherwise I would have to move the consolidation under all like flakes.all.github

Anyways, the user experience is still growing. This idea is pretty naescent so I’m learning a bit as I go. I’m happy you are enjoying it!

2 Likes

This is a great idea. Only thing I would add is the naming is a bit long-winded.

Maybe a simple OFL - instead of omniflake in all related nix files would be nicer

You have to add omniflake as a flake input, so you can give it a name of your liking. inputs.ofl = { url = "github:fzakaria/omniflake"; ... }; leads to inputs.ofl.flakes.home-manager path, for example.

3 Likes

I just setup this as an expansion to the dendritic pattern.
With just 2 inputs and a shallow merge I have flake-parts, import-tree, disko, agenix, nixvim, etc.
Now I can just use a flake input from anywhere without having to declare it.

This is probably a horrible idea, I have some follows logic I know I need to fix but this is a very clean flake.

{
  description = "NIX";

  inputs = {

    nixpkgs.url = "github:nixos/nixpkgs/release-26.05";
    omniflake = {
      url = "github:fzakaria/omniflake";
      inputs.nixpkgs.follows = "nixpkgs";
    };

  };

  outputs = nativeInputs: let inputs = nativeInputs.omniflake.flakes // nativeInputs ; in
    inputs.flake-parts.lib.mkFlake {
      inherit inputs;
      specialArgs = {
          mylib = import ./libs/lib.nix { lib = inputs.nixpkgs.lib; };
          mvpkgs = inputs.nixpkgs-multiverse.multiverse.x86_64-linux;
        };
    } (
    inputs.import-tree [
      inputs.home-manager.flakeModules.home-manager               # Import home-manager flake-parts module
      (inputs.import-tree.matchNot ''.*/secrets\.nix'' ./modules) # Import everything except secrets.nix
    ]
  );
}