Confusion on overlay usage

Hi,

I’m still in my painfull journey through configuring my asahi-linux NixOS.
Today I’m trying to add a non-nixpkgs gtk theme to my configuration as an overlay.

I followed theses guidelines on overlays but nothing I try seems to work.

No mater what I do, I get an error when rebuilding.

Here is the first thing I tried:

flake.nix relevant parts:

 asahi-mac = lib.nixosSystem {
          system = "aarch64-linux";
          modules = [
            # This is the line I'm not sure of, but no matter what I write here, I get an error
            (args: { nixpkgs.overlays = import ./overlays args; })

            ./hosts/asahi-mac
            home-manager.nixosModules.home-manager
            {
              home-manager.useGlobalPkgs = true;
              home-manager.useUserPackages = true;

              home-manager.extraSpecialArgs = {
                myAwesome = myAwesome;
                nixpkgs-unstable = import nixpkgs-unstable {
                  system = "aarch64-linux";
                };
              };
              home-manager.users.jm445 = import ./home/asahi-mac;
            }
          ];
          specialArgs = {
            myAwesome = myAwesome;
            nixpkgs-unstable = import nixpkgs-unstable {
              system = "aarch64-linux";
            };
          };
        };

overlays/default.nix:

args:
# execute and import all overlay files in the current
# directory with the given args
builtins.map
  # execute and import the overlay file
  (f: (import (./. + "/${f}") args))
  # find all overlay files in the current directory
  (builtins.filter
    (f: f != "default.nix")
    (builtins.attrNames (builtins.readDir ./.)))

I actually don’t really understand how this file works. I understood that there is some sort of reading of all subdirectories to read their default.nix, but the map part is obscure to me.

overlays/GTK-Midnight/default.nix

{ config, lib, pkgs, ... }:
(self: super: {
  gtk-midnight = super.callPackage ./../../pkgs/GTK-Midnight {};
})

When I rebuild with this configuration, I get an error about the overlay definition: “function ‘anonymous lambda’ called without required argument ‘pkgs’”

Then as I don’t quite understand the behavior of the default.nix that is loading all overlays, I tried getting rid of it by referencing directly the required overlay:

(args: { nixpkgs.overlays = [ import ./overlays/GTK-Midnight ]; }) # Not sure of this syntax

This doesn’t work either: “A definition for option nixpkgs.overlays is not of type nixpkgs overlay’”

I can’t actually debug anything because I don’t understand what I’m doing.
Why should I declare overlays as modules ? What type should a module be in the module list ?

The nix language is driving me crazy. I’m in the process of learning it since more than 6 months but I’m still completely lost in the syntax and basic concepts of it, trying to use code I find on the net without understanding it.

Thank’s for any help,
JM445

1 Like

Ok, so there are a lot of things going on here. I think these are the issues:

  1. You’re very confused about the NixOS module system and how it works
  2. You’re severely over-engineering your overlays because the flakes book recommends something way over-the-top for your use case
  3. You’re unnecessarily trying to write an overlay system to add a package to your nixosSystem
  4. You seem to have some kind of invisible UTF character in your args list or a non-standard argument in a derivation you’re callPackage-ing
    • or the flakes book’s example is wrong, but I think their mess is barely still correct

Let’s start with 4. You say this is the error:

Note it says 'pkgs? Did you write 'pkgs instead of pkgs in ./../../pkgs/GTK-Midnight/default.nix?

If not, is there some weird character in front of the pkgs in overlays/GTK-Midnight/default.nix?

The code as you’ve written should be working otherwise. But you’re desperate, so let’s simplify.


Next up, 2. If you really only have this simple of an overlay, it’s completely overkill to go through all that effort. Just add this to your host config:

# ./hosts/asahi-mac/default.nix
# If you already use arguments for this module, keep them of course.
{ pkgs, ...}: {
  nixpkgs.overlays = [
    # Using prev: final: because that's a much clearer abstraction when
    # you know what overlays do
    (prev: final: {
      # Using final because we want to use `callPackage` from the *final*
      # nixpkgs, i.e., the one that comes out after all overlays were applied
      gtk-midnight = final.callPackage ./../../pkgs/GTK-Midnight {};
    })
    # You can add more overlays here, if you like
  ];

  environment.systemPackages = with pkgs; [
    gtk-midnight
  ];

  # Other contents of the file mostly don't matter for this example
}

Assuming your ./../../pkgs/GTK-Midnight/default.nix file isn’t inherently broken, this should work.


Next up, 3. AIUI, you’re using an overlay to create pkgs.gtk-midnight. This is not necessary - there’s no need to register your downstream package with pkgs at all. Let’s just add your package to environment.systemPackages and skip the overlay:

# ./hosts/asahi-mac
{ pkgs, ... }: {
  environment.systemPackages = [
    (pkgs.callPackage ./../../pkgs/GTK-Midnight {})
  ];
}

Since you’re using a flake, you could go a step further and create self.packages.${system}.gtk-midnight, and then do something like this:

{ pkgs, self, ... }: {
  environment.systemPackage = [
    self.packages.${pkgs.system}.gtk-midnight
  ];
}

That would be by far the cleanest implementation. Happy to help with that if you need a hand.


Then, 1. 1 is more complex. I’ve explained this a couple of times over the years, but I can’t seem to find my big long posts on the topic, so here we go again:

Fundamentally, NixOS modules (i.e., the stuff you’re supposed to pass to modules up there) are just attrsets (read: fancy JSON objects, or the stuff between {}). NixOS takes all modules you define, and combines them into one big attrset, which it then uses as input to write a script that makes the system you defined.

Modules can be written like this:

{
  imports = [
    # Paths of other modules that should be merged into this one here
  ];
  options = {
    # Custom option definitions here
  };
  config = {
    # Settings for defined options in this or other modules here
  };
}

Alternatively, you can skip the options definition, and only specify configuration settings:

{
  imports = [
    # Paths of other modules that should be merged into this one here
  ];
  # Settings for defined options in this or other modules here
}

NixOS will also pass arguments to any modules that are defined as functions:

# Note the ellipsis (`...`) just means "don't error out if other arguments
# are given"
#
# NixOS will put any arguments in `_modules.args` in your module's function
# arguments. This includes things passed in via `specialArgs`, but also some
# default things like `pkgs` and `config`.
#
{config, lib, pkgs, ...}: {
  imports = [
    # Paths of other modules that should be merged into this one here
  ];
  # Settings for defined options in this or other modules here
}

This is all there is to NixOS modules.

This is importantly very different from derivations (package definitions) or overlays. NixOS modules are their own little mini-ecosystem in the nix world, so their syntax and conventions are very different.

This is why the following won’t work:

If I rewrite that to look more like a module it’s something like this:

# We've replaced the attrset input with specific attrset components with a
# pure `args` variable we can't look inside of, but it's still given an
# attrset with the contents of `_module.args`
args: {
  nixpkgs.overlays = [
    import ./overlays/GTK-Midnight
  ];
}

Note that your overlays/GTK-Midnight/default.nix file is also a module. You’re import-ing it, which will make nix place the contents right in there:

# We've replaced the attrset input with specific attrset components with a
# pure `args` variable we can't look inside of, but it's still given an
# attrset with the contents of `_module.args`
#
# We could also write:
#
# { config, lib, pkgs, ...} @ args: {
args: {
  nixpkgs.overlays = [
    (
      { config, lib, pkgs, ... }:
      (self: super: {
        gtk-midnight = super.callPackage ./../../pkgs/GTK-Midnight {};
      })
    )
  ];
}

Obviously this doesn’t work, because nixpkgs.overlays wants a list of overlays, not a list of modules.

We could make it work if we wanted to, by calling the first level of functions with the args input (and thereby “consuming” the first level of indirection and giving nixpkgs.overlays a function it knows what to do with):

args: {
  nixpkgs.overlays = [
    ((
      { config, lib, pkgs, ... }:
      (self: super: {
        gtk-midnight = super.callPackage ./../../pkgs/GTK-Midnight {};
      })
    ) args)
  ];
}

But you’re not even using the config, lib, pkgs in your overlay, so I propose we just write a sane overlay instead and forget about the args:

{
  nixpkgs.overlays = [
    (self: super: {
      gtk-midnight = super.callPackage ./../../pkgs/GTK-Midnight {};
    })
  ];
}

Honestly IMO the flakes book you’re reading is making a huge mistake trying to turn overlays into quasi-modules by passing the module args into them. It’s no wonder you can’t keep the two concepts apart when they’re forcibly mixing them. Not to mention that unreadable auto-file-import function.

I’ll probably stop recommending that book, this is just insane - no newbie should reasonably be expected to deal with this.

Hello,

Many thanks to you to take so much time and effort to explain me all of this.

It is now a bit clearer in my mind.

For the 4th point, I have searched for a strange UTF character in my files but didn’t find any. However I have been able to fix the rebuild by removing the config, lib and pkgs arguments of the overlay module, just keeping { … } as I don’t use them in the overlay.

But anyway, you’re right I may be overthinking this, I will try the approach you describe with callPackage inside my modules.
Actually, I tried the overlay approach because I made something different for another package that I made as a sub-flake. This was not working very well as the flake is referenced from local path, I needed to manually remove from the lock file every reference to it when rebuilding from different hosts.

However I don’t quite understand what you mean by creating packages.${pkgs.system}.gtk-midnight. Does that mean I must add an input to my flake ?

I understand better how modules works but I still miss a point:
You say that import makes nix place the module content where it’s used. Does that means that the code is simply loaded, a bit like a C #include or is the function in the module called with its parameters given automatically by nix ?

Lastly a syntax question, in the following snippet I don’t understand the use of the parenthesis around the overlay:

nixpkgs.overlays = [
    (self: super: {
      gtk-midnight = super.callPackage ./../../pkgs/GTK-Midnight {};
    })
  ];

I understand that nixpkgs.overlays expects a list of overlays, and overlays are simply “functions of one arguments, that returns another function of one argument, that returns an attrset”. Are the parenthesis used only to “wrap” the function or do they have a special effect I don’t get ?

Anyway, again, big thanks for your patience.
JM445

A bit like a C #include. The reason it might seem like the function is called is because people typically put the arguments right next to the import like here:

Of course, the filename is not evaluated yet there, it might be easier to see if we simplify the expression to a specific one:

import ./GTK-Midnight/default.nix args

So yeah, we’re passing args to the function that is in that file, but not because nix automatically evaluates it with mystery arguments.

This is very different from the imports attribute in NixOS modules - those are automatically merged into the set of modules being evaluated, and given the module arguments automatically.

Pretty much.

In general nix lists are prone to splitting function calls into lists for example like this (not valid nix, commas added to visualize what nix does):

[
  super.callPackage,
  ./../../pkgs/GTK-Midnight,
  {}
]

It’s interpreted that way because functions and their arguments can safely be interpreted as variables, and there is no list element separator.

In this specific case, I’m not sure if self: super: would confuse the nix parser (since they aren’t valid expressions by themselves), but better safe than sorry.

No, I’d suggest adding another output to the flake. Something like this:

# flake.nix
# inputs are irrelevant
outputs = { nixpkgs, self, ... }: let
  # If you have more architectures, use something like flake-utils
  system = "aarch64-linux";
  pkgs = nixpkgs.legacyPackages.${system};
in {
  packages.${system} = {
    gtk-midnight = pkgs.callPackage ./pkgs/GTK-Midnight {};
  };

  # Simplified for brevity
  nixosSystems.asahi-mac = lib.nixosSystem {
    system = "aarch64-linux";
    modules = [
      ./hosts/asahi-mac
    ];
    specialArgs = {
      # Nicer than self = self;
      inherit self;
    };
  };
}

This would give your modules a new self argument, which will just contain all the outputs of your flake - including packages. So you could then write in any module:

{ self, pkgs, ... }: {
  environment.systemPackages = [
    # We need to specify the architecture for flake packages,
    # so we use ${pkgs.system} to get the correct one for the system.
    #
    # This is nice if it changes or we reuse the module somewhere
    # else.
    self.packages.${pkgs.system}.gtk-midnight
  ];
}

Also, unrelated, but a collorary of my explanation around modules and imports is that you can put all that home-manager gunk in a separate file too:

asahi-mac = lib.nixosSystem {
  modules = [
    ./hosts/asahi-mac
    ./home/asahi-mac/nixos-entrypoint.nix
  ];

  specialArgs = {
    # If you collect all inputs with an @ inputs, you could even put that
    # here and automatically propagate all your flake inputs to your
    # module args
    inherit self home-manager myAwesome;

    # import-ing this is an anti-pattern, by the way - only necessary if
    # you need an overlay
    nixpkgs-unstable = nixpkgs-unstable.legacyPackages."aarch64-linux";
  };
}
# ./home/asahi-mac/nixos-entrypoint.nix
{ home-manager, myAwesome, nixpkgs-unstable, ...}: {
  imports = [
    home-manager.nixosModules.home-manager
  ];

  home-manager.useGlobalPkgs = true;
  home-manager.useUserPackages = true;

  home-manager.extraSpecialArgs = {
    inherit myAwesome nixpkgs-unstable;
  };

  home-manager.users.jm445 = import ./asahi-mac;
}

I wish more people did this, it makes flakes so much less cluttered. Even nix code can read like lasagna!

Don’t want to hijack the thread, so could move to a different one, but I thought I’d suggest raising the issue on Issues · ryan4yin/nixos-and-flakes-book · GitHub or suggesting to @ryan4yin to make some adjustments. I find that book very useful, as do I find your config and posts @TLATER, so would be cool to see the book benefit from any of your input/suggestions!

The book only recommends looping over files like that if you have so many overlays that it becomes hard to manage. Maybe can further explain how the code actually works, or move to an advanced overlays page to determine newbies, etc.

2 Likes

Yeah, I think it’s also worth explaining on that page when overlays are not actually needed. I should make some suggestions at some point, indeed.

2 Likes

I learned this pattern from nix-community/nur-packages-template/overlay.nix.
Because this is a nix-community project, I always thought it was a best practice.

But as you said, it is really not beginner-friendly, I will find time to adjust this part of the content.


I’m not an expert on NixOS, and I’ve only been using NixOS for less than 9 months, so I think there must be some (insane) misconceptions or complex cases like this in the book. If anyone finds anything incorrect, just let me know about it, I’ll fix it.

The reason why I wrote this little book was only because no one in the community did it for me, who was a beginner at the time, so I chose to do it myself.
Even though I knew I could make mistakes, it was much better than do nothing.

2 Likes

I was somewhat too harsh in my criticism. This came from my initial glance at it looking like it’s a safe thing to just throw at newbies, so seeing that it’s not that clear-cut made me realize I should have looked closer.

Having something is important, and before you there was no good resource at all. I should just contribute where I see issues, and make it safe to throw around :slight_smile:

nur templates come from a different age, when there were few options besides overlays to modify packages. Most modules didn’t have .package options yet. NUR has a very different target audience too. The function is fine, this just isn’t the right place for it IMO.

2 Likes

You’re right, I just removed the content related to modular overlays, as it’s not necessary(and too complex) for beginners.