Flakes for multiple machines with nonfree packages and modules

Help
1

I’ve been banging my head on this for a week or so, and I wanted to make sure to document it so others wouldn’t bang their heads…

I’ve got a situation where I’m using a flake.nix to describe several machines. Some of these machines should use nonfree packages. As described variously online, nonfree packages in a flake means you must reimport the nixpkgs with a new config because the one imported by the flake described in “inputs” doesn’t have nonfree enabled. I got that far by searching, the basic structure is something like this:

{
  description = "A basic NixOS flake";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
#    nixpkgs.config.allowUnfreePredicate = pkg : true;
  };

  outputs = {self, nixpkgs, ...}@inputs: 
    let pkgs = import nixpkgs { system = "x86_64-linux"; config = {allowUnfree = true; }; lib=nixpkgs.lib; }; in 
      let 
        hostfuns = ( (import functions/hostfuns.nix) {inherit nixpkgs pkgs;}); in
        
        {
          nixosConfigurations = {
            machinea = hostfuns.desktop "machinea";
            machineb = hostfuns.desktop "machineb";
...
          };
        };

}

Ok, so hostfuns is basically a file where I create some functions for making different “types” of machines. In this case the examples “machinea” and “machineb” are configured using the “desktop” function and that’s passed the name of the machine so that machine-specific modules can be loaded.

the desktop function inside hostfuns looks something like this:

{nixpkgs, pkgs} : {
...

  desktop = hostname :
    nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      # specialArgs = {inherit pkgs;};
      modules = [
        ../base/configuration.nix
        ../desktop/desktop.nix
        ../desktop/homenetwork.nix
        ../host/${hostname}.nix
      ];
    };
}

So, with that line commented out about specialArgs, even though I’ve imported nixpkgs with the config that allows nonfree, things don’t work… It took me a while to figure this out. but when you construct the nixosSystem object and tell it what modules to use, eventually it has to call those modules with some arguments, and you need some way to tell it to call the module with pkgs that corresponds to your non-free package object. So that’s what specialArgs argument to nixosSystem constructor is for.

I found this in the nixos and flakes book.

So far it actually works… but when I do the build I get a message that I don’t understand, perhaps there’s some additional subtlety about how this should be done?

evaluation warning: You have set specialArgs.pkgs, which means that options like nixpkgs.config
                    and nixpkgs.overlays will be ignored. If you wish to reuse an already created
                    pkgs, which you know is configured correctly for this NixOS configuration,
                    please import the `nixosModules.readOnlyPkgs` module from the nixpkgs flake or
                    `(modulesPath + "/misc/nixpkgs/read-only.nix"), and set `{ nixpkgs.pkgs = <your pkgs>; }`.
                    This properly disables the ignored options to prevent future surprises.

I’ve been using Nix/NixOS for about 3 weeks, so the fact that I’ve got this far is quite remarkable to me, but I have literally no idea what this message is telling me, and I can’t seem to web-search up any readOnlyPkgs documentation or know if this warning is a real problem or not etc.

Hope this information is helpful to someone in the same boat, and/or that someone can explain that message or if I’m doing something wrong here.

Thanks to everyone in the community for making this stuff possible. Gonna make administering 20 odd machines much easier!

2

I appreciate the determination in wrangling an unfamiliar programming language with an unfamiliar ecosystem, but you’re overengineering and making some notable mistakes (one of which is the cause of that message). I fully understand how you got here, though, these entrypoints are confusing, not very well designed from an API perspective because they’re a bit of a crutch until things settle, and flakes themselves are an underdocumented, experimental feature.

I’ve written functions like this myself, and also struggled to understand specialArgs initially :slight_smile:

Firstly, dealing with the boilerplate for cattle-style machines.

If you have multiple machines that should be configured identically, just define one nixosConfigurations entry for all of them, and build those machines with:

nixos-rebuild boot --flake <dir>#<entry>

This will pick the specific entry from the nixosConfigurations - that way you don’t need to define dozens of the same nixosConfigurations entry to begin with, and you can stop farming out the rest of the definition to a separate function definition (though, depending on how many “classes” of devices you have, a separate function to turn 3 lines into 1 may still be appropriate).

Secondly, if you’re going to have per-host modules, you might as well put all the imports in them:

# ../host/${hostname].nix
{
  imports = [
    ./base/configuration.nix
    ./desktop/desktop.nix
    ./desktop/homenetwork.nix
  ];
}

That way you only need to put a single file in modules, which significantly cuts down on boilerplate in flake.nix - though this comes down to style, admittedly, I don’t dislike making it explicit that a specific machine contains a desktop.nix (so that I know it is supposed to be an end-usery device).

Between these, I don’t think there are many projects where it makes sense to have a custom nixosSystem function. I anyway think it rarely makes sense to have that, especially if you’re going to move the definition outside flake.nix, because what even is the purpose of using flakes for nixosConfigurations if you’re going to throw away its functionality as a project index?

Secondly, specialArgs and inherit pkgs. This is… a pretty big mistake. It will result in options not working at best, evaluation breaking completely at worst.

This replaces the pkgs in the { pkgs, ... }: you have at the top of every module with whatever you pass in there. The NixOS module system however expects to instantiate its own version of pkgs from the nixpkgs that calls nixpkgs.lib.nixosSystem, and it makes specific modifications to it (usually though the nixpkgs options in the module system, but not always).

By overwriting this with your own custom definition, you’re basically completely breaking the NixOS module system.

You do this because:

But you actually don’t! Since the NixOS module system manages its own pkgs arg, you can tell the NixOS module system to allow unfree packages. You just need to set:

# base/configuration.nix
{
  nixpkgs.config.allowUnfree = true;
}

Or you can even set it to a function to limit your unfree packages to a specific set, if you care for that:

# base/configuration.nix
{
  nixpkgs.config.allowUnfreePredicate = pkg:
    builtins.elem (lib.getName pkg) [
      "hello-unfree"
    ];
}

Your attempt at solving the problem originates from another misunderstanding. In the flake world, writing import nixpkgs { } in any context is an anti-pattern. You should never do that, and instead use nixpkgs.legacyPackages.${system}* where you need immediate access to packages in a flake.nix, or use nixpkgs.lib for library access.

For the purpose of a flake that does nixosConfigurations, you should never, ever directly refer to your nixpkgs input, except to call nixpkgs.lib.nixosSystem.

Honestly, for a project that only manages NixOS configurations, I would even discourage you from using flakes in the first place, they’re just a glorified npins + project index in this context. But that’s an aside…

* exception

The exception is if you deliberately instantiate a copy of a different nixpkgs version. Since flakes have no way to apply configuration to their inputs, and the NixOS module system does not apply to this second instance, you need to do something else.

There’s a project for this scenario, though: GitHub - numtide/nixpkgs-unfree: nixpkgs with the unfree bits enabled

It’s not; it exists to pass additional variables into the module arguments. This is useful if you have more flake inputs, or want to set actual variables (e.g., I’ve seen people define “main” system user’s usernames that way).

specialArgs should only be used sparingly, those two things are honestly the only legitimate use cases I’ve seen so far. It most certainly should not be used to override existing args - like pkgs - in the NixOS module system, though.

Even then, the _module.args option is more appropriate for usernames, or any other variables that might fit a similar purpose. IMO specialArgs should practically only be used to funnel flake inputs into the module system.

3

Thank you so much for your extensive response. I posted here so that this whole process would be less opaque for me and for others and I appreciate your help!

why the gaggle of functions

First, the reason for my gaggle of functions… None of my machines are true cattle. At a minimum of course they have their own hostname. But also for example there are commonality for all the desktops in my home, but my kids should have minecraft and whatnot, and my desktop should have vscodium, and the media-room PC should have Ardour and such… All of them should have systemd-networkd based network settings, but each one should have a different static IP and ipv6 token. There might be some special firewall stuff, whatever. So thats why there’s kind of a major structure “it’s a desktop” or “it’s a laptop” but also host-specific customizations “here’s its IP, here are a few custom packages it needs, etc”

The purpose, for me, of using flakes for nixos configurations is that I have all the machines defined in a single pure definition that I can push out from a centralized build VM. When I want to change how some machines work, I adjust the config on my laptop or desktop, check it out on the VM, and push it to the machines that need updating. This way I’m not logging in and checking out the config on each machine, wasting a lot of keyboard time.

specialArgs and import nixpkgs

There are a number of things I don’t understand, and there are a number of things that will absolutely come up when you web-search for stuff like “nixos flakes unfree packages”. When I do that search now on Kagi, on the first page I’m gonna get:

https://stackoverflow.com/questions/77585228/how-to-allow-unfree-packages-in-nix-for-each-situation-nixos-nix-nix-wit

and I’m also gonna get a bunch of similar results from other sites, and essentially all of them are going to result in the recommendation

let pkgs = import nixpkgs {config.allowUnfree = true;}
...

The argument given in most of these web pages is that nixpkgs is imported by the flakes machinery as part of the input = of the flake, and so it’s already been configured and you somehow can’t change nixpkgs.config.allowUnfree without re-importing.

This may be wrong, but it is absolutely the dominant explanation in the searchable sites that discuss how to use unfree packages in a flake.

I think you’re saying that you can do something like:

modules = [ ./foo.nix ./bar.nix ./baz.nix ];

in the call to nixosSystem and as long as one of those has nixpkgs.config.allowUnfree = true; then you can use unfree packages. I can tell you that that isn’t true in my experience. I absolutely had nixpkgs.config.allowUnfree = true; in my base/configuration.nix and yet, I was getting unfree packages errors. (I just checked, it’s still there but commented out, but when it was enabled it did NOT help with the unfree packages issue).

However, I’ll have to test it again and perhaps it’s on a per-module basis? I think I may have set config.allowUnfree in one module but then tried to include nonfree packages in a different module, and the config doesn’t translate across between modules? Was that the issue?

Thanks again for your help!

4

Fair enough, then ultimately this comes down to code style. My personal preference is still to make the nixosSystem calls in nixosConfigurations, at best factoring the system and specialArgs definition into a separate function. If the modules are declared separately, you cannot see what the difference between the systems is, and you end up with just a list of hostnames.

If you prefer a more implicit style, personally I would go a step further and simply iterate over files, taking the hostnames from the filenames to build the attrset. I’m not usually a big fan of using the filesystem as a code artefact, though, unless you’re dealing with a truly huge codebase that has a lot of churn. I’d just pick one or the other over a half-measure :wink:

That’s what you’re using NixOS for. Flakes are completely optional for this purpose and needn’t be involved at all.

It’s a bit of a misconception that you can’t have multiple NixOS definitions in one project without a flake. If you check out your repo and run e.g.:

NIX_PATH="$NIX_PATH:nixos-config=$PWD/host/$hostname.nix" nixos-rebuild --target-host $hostname

you end up with the exact same result, without ever evaluating flake.nix - with your current project (well, after consolidating the modules into a proper per-host entrypoint so instead of modules you use imports). It might even be faster, depending on how eval cache vs. not copying the world pointlessly pans out.

Flakes do offer a “purity” check as well, which just boils down to disallowing imports from absolute paths. Just… not doing that is pretty easy, and from the sound of it your workflow already makes that impossible, so you don’t need this check.

I’m not telling you to abandon flakes, I’m just trying to clear up the misconception that flakes are doing anything impactful for you. For NixOS projects specifically it’s little more than a style guide (which most folks - including you - feeling NixOS out explicitly refuse to follow anyway).

Yep, and IME stackoverflow is just slightly better at giving actually good advice than your average LLM. Not surprising given the latter is effectively an index of the former, to be fair.

Not your fault, also not really their fault, as I said, the docs aren’t great and using flakes makes things harder for you (and everyone talking about flakes on stackoverflow) because the API exposed for it in the NixOS world is just as experimental and underdocumented/developed as the flakes concept itself. It’s no wonder random drive-by contributors to a really generic forum have no idea how any of this works, the official docs don’t even mention the existence of nixosSystem, let alone explain how it should be used.

Search engines, SEO, confirmation bias, and the ever-increasing centralization of the internet to like 10 walled-garden platforms do the rest, and we end up here with you getting a warning and telling me I’m wrong about how to fix it because stackoverflow said so. Sigh.

Just a small vent, read with a largely sarcastic tone - you’re really good about asking questions and communicating your understanding, I appreciate this.

I can tell you that that is actively true in my configuration today, and has been for over half a decade!

But you don’t have to take my word for it, you can also read the source code for yourself: nixpkgs/nixos/modules/misc/nixpkgs.nix at c7ab75210cb8cb16ddd8f290755d9558edde7ee1 · NixOS/nixpkgs · GitHub

Most likely it’s being overridden by the pkgs you pass into specialArgs, or you’re not actually evaluating the module, or any number of mistakes you could be making. I’m not going to try guessing any further without actually seeing code, though.

It’s not, see the code I linked and how it flows into eval-modules.nix, the evaluated pkgs ends up in _module.args, which is globally passed to all modules.

specialArgs is just passed into _modules.args which means it overrides the exact same pkgs the nixpkgs option does. In fact there are explicit carve-outs to permit passing in a custom pkgs, which is why what you’re doing even works, but this is definitely not using the module system as intended.

5

Awesome, thank you for your explanation and patience. I’ll branch my config and refactor it and see if I can make it work! Also, I’m definitely not telling you you’re wrong, only explaining why and how I came to the surface level understanding that I had and tried the things I tried (and it’s not just stackoverflow that said the same thing, lots of people have probably repeated this anti-pattern)… All of this would be much easier to figure out if there were more online resources that explained it… which is why you and I are having this conversation to make sure there’s something to search up and see an alternative.

:wink:

6

Yeah :frowning: It’s the source of my frustration. I spend a lot of time on thoughtful responses around here, but the cargo cult is winning. I feel like I reply to someone wth a variant of this question at least once a week.

Thanks for the positivity; If nothing else, the adoption of NixOS increasing is a nice thing. I just hope bad experiences resulting from the flood of poor advice don’t make this a short-lived trend.

7

Ok, so I branched my config, removed the import nixpkgs {config.allowUnfree=true;} stuff, and the specialArgs stuff, put

nixpkgs.config.allowUnfree = true;

into base/configuration.nix and made sure I was importing the base/configuration.nix module and for sure it DID work even with the unfree package spotify specified for the laptop I’m testing on.

So, indeed at some point I probably tried this in some incorrect way and it wasn’t working not because it doesn’t work, but because my code was broken.

Thanks for your help. I’ll summarize it here and mark this solved!

{
  description = "A basic NixOS flake";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
  };

  outputs = {self, nixpkgs, ...}@inputs:  
      let 
        hostfuns = ( (import functions/hostfuns.nix) {inherit nixpkgs;}); in
        
        {
          nixosConfigurations = {
            machinea = hostfuns.desktop "machinea";
            machineb = hostfuns.desktop "machineb";
...
          };
        };

}

{nixpkgs} : {
...

  desktop = hostname :
    nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ../base/configuration.nix
        ../desktop/desktop.nix
        ../desktop/homenetwork.nix
        ../host/${hostname}.nix
      ];
    };
}

and inside base/configuration.nix

  nixpkgs.config.allowUnfree = true;

or you could use the predicate version to filter the allowed list of packages.

1 Like