From NixOS to NixOS flake

Hello, I am not an expert, I am learning. So this question is probably basic.

I put together a gist to reproduce where I am so far. I would like to build NixOS for netbooting with flake.

here the gist (https://gist.github.com/gianarb/c72c5158b237749c365183892b3fcce6) that you can clone with:

git@gist.github.com:c72c5158b237749c365183892b3fcce6.git

This is the error I get:

$ nixos-rebuild build --impure --flake .#generic
building the system configuration...
warning: Git tree '/home/gianarb/.dotfiles' is dirty
warning: creating lock file '/home/gianarb/.dotfiles/nixos/machines/lab-generic-netbooting/flake.lock'
warning: Git tree '/home/gianarb/.dotfiles' is dirty
error: You're trying to declare a value of type `string'
       rather than an attribute-set for the option
       `system'!

       This usually happens if `system' has option
       definitions inside that are not matched. Please check how to properly define
       this option by e.g. referring to `man 5 configuration.nix'!
(use '--show-trace' to show detailed location information)

I am not sure what this issue tells me. I would also like to remove needs for --impure but I am not sure how to do so.

Any pointers for me?
Thanks a lot

3 Likes

Good thing you’re posting in the “learn” category then :wink: Let me just review your code and point out all the small things, we’ll fix the errors along the way.

Warning: Turns out there’s a lot of strange things happening here. Apologies for the wall of text, I’ll very happily answer more specific questions, too.


Let’s start with the flake.nix:

{
  description = "A very basic flake";

  inputs =
    {
      nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.05";
    };

  outputs = { nixpkgs, home-manager, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        config = { allowUnfree = true; };
      };
      lib = nixpkgs.lib;
    in
    {

      nixosConfigurations = {
        generic = lib.nixosSystem {
          inherit system;
          modules = [
            ./configuration.nix
          ];
        };
      };
    };
}

This looks mostly good, but there are a few things to explain. Firstly, the nixpkgs you use here will not have the allowUnfree config option set. This is because you’re using nixpkgs.lib, which refers to the un-doctored nixpkgs input, rather than the imported version of it.

To understand what I mean, let me explain what happens here:

pkgs = import nixpkgs {
  inherit system;
  config = { allowUnfree = true; };
};
lib = nixpkgs.lib;

In the first line, you tell nix to import the default.nix file in the directory of nixpkgs, which refers to the place where nix downloaded your nixpkgs input to.

That is, you will be evaluating this file from the nixpkgs repository, and assigning the resulting value to the pkgs variable.

After that, you tell nix to assign the lib output from the nixpkgs input to the lib variable. That is, you take this line from the nixpkgs flake, and reassign it to lib.

Later you then go on to use lib to build your configuration, so pkgs is actually completely unused, and your configuration doesn’t actually apply.

An easy fix is simply to use pkgs.lib instead of nixpkgs.lib. Any evaluation of nixpkgs will always include the lib attribute you’re used to.

With flakes, however, it’s better to avoid using import if you can, because then we can make use of the flakes’ evaluation cache and don’t have to evaluate nixpkgs every time. This is more important for flakes that will be used as dependencies in other flakes, but I like doing it in other flakes if I can get away with it easily.

In this case you probably can:

{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.05";
  };

  outputs = {
    nixpkgs,
    home-manager,
    ...
  }: let
    system = "x86_64-linux";
  in {
    nixosConfigurations = {
      generic = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          ./configuration.nix

          # This is an inline module declaration, they work just like
          # `configuration.nix` :)
          ({...}: {
            # For bonus points, move this into configuration.nix
            # instead of specifying it in your flake.nix
            nixpkgs.config.allowUnfree = true;
            # For bonus bonus points, use `allowUnfreePredicate`
          })
        ];
      };
    };
  };
}

If you really want to you can of course still do the lib = nixpkgs.lib thing, but it’s simply renaming a short variable you only use once.

NixOS has a nixpkgs.config option, and you can simply put the configuration in there, instead of importing nixpkgs and modifying its configuration there. This avoids evaluating nixpkgs multiple times :slight_smile:

This only works for the nixpkgs whose lib you use for nixosSystem, so if you have two different nixpkgs inputs you will probably need to change the configuration of that one the way you did. But you would use that quite differently; feel free to come back if you ever need a hand figuring it out.


On to your configuration.nix:

pkgs.symlinkJoin {
  name = "netboot";
  paths = with bootSystem.config.system.build; [
    netbootRamdisk
    kernel
    netbootIpxeScript
  ];
  preferLocalBuild = true;
}

Surprise! This is not a NixOS configuration at all, you’re writing a derivation that builds a NixOS configuration.

Is this your intention? If not, shout, but for now I’ll assume you’re using this as an ad-hoc “install CD” where you just dd those files into a file system somehow and it actually boots with the NixOS system you configured, and that you then want to use nixos-rebuild with this config to update.

I’ll be frank; I have not yet experimented with this kind of setup, so I don’t know if this will work. If I was going to do this I would probably use nixos-generators instead.

But I think my advice in this case would be as follows:

  1. Split out the NixOS configuration as an actual nixosConfiguration output.
  • This is useful because you can then use the NixOS ecosystem as you usually would, and for example test your system with nixos-rebuild build-vm.
  1. Make a second packages output that does this symlinkJoin on the nixosConfiguration output.
  2. Build the package instead of the NixOS configuration when you want your image, and build the NixOS configuration if you want to update to it on the running host.

Let me explain.


Firstly, nixosConfiguration doesn’t just take an arbitrary nix expression and magically turn it into a configuration you can build with nixos-rebuild. In your flake.nix, you use the lib.nixosSystem function to build a NixOS configuration. This sets up the module system and makes the build actually work, and creates a derivation that consists of the activation script for your config.

In the modules list, you are supposed to give it a list of NixOS modules (a set of functions taking config, lib, pkgs, ... and returning a set of attrs that match up to configuration options), so that the function can glue them together into a configuration. What you are giving it is a derivation, i.e. instructions on how to build a store path; This cannot work, the types don’t match.

So instead, you probably want to make configuration.nix contain only the stuff you assign to configuration in your bootSystem attrset, something like this:

# configuration.nix
{ config, pkgs, lib, ... }: with lib; {
  imports = [
    <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix>
  ];

  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEKy/Uk6P2qaDtZJByQ+7i31lqUAw9xMDZ5LFEamIe6l"
  ];

  ## Some useful options for setting up a new system
  services.getty.autologinUser = mkForce "root";

  #<snip>
}

Once you have this, you can build your system with nixos-rebuild build --impure --flake .#generic (we’ll get to impurity later), and it will output the files in result. But it won’t be exactly like your current config.

Next, we need to create a new output in flake.nix so that we can package up this system like you used to. Add a new packages output as follows:

packages.${system}.netboot = nixpkgs.legacyPackages.${system}.symlinkJoin {
  name = "netboot";
  paths = with self.nixosConfigurations.generic.config.system.build.topLevel; [
    netbootRamdisk
    kernel
    netbootIpxeScript
  ];
  preferLocalBuild = true;
};

Note the reference to self - you will need to add self to the arguments of the outputs function. self here refers to the flake itself, and you can refer to outputs of the flake like this - in this case the system we built with lib.nixosSystem.

I’m not sure this will work - I don’t remember in detail how importing nixpkgs differs from using lib.nixosSystem. But if it does work, you should be able to now build your system with nix build --impore .#netboot.


Finally, on impurity. In your configuration.nix you have a couple of statements like this one:

<nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix>

The <> syntax means “look up the first element in this path in the $NIX_PATH variable to see where it is on my disk, and replace it with that path”.

This is obviously impure, because a) it refers to a file outside of the flake directory and b) because it relies on an environment variable. Both of these things will only be present on the device you write the code on, so it’s not reproducible. Hence, using this syntax is practically forbidden in pure mode.

Instead, we need to use our flake inputs to identify where these files are, and pass down the information into our configuration. This is not impure, because the flake inputs have tracking information (specifically, checksums calculated and written to flake.lock) to ensure that the inputs are always the same, and so that we know how to get them.

For modules that are part of nixpkgs, there is a slightly less well-known module input called modulesPath. We can simply use it:

# configuration.nix
{ config, pkgs, lib, modulesPath, ... }: with lib; {
  imports = [
    (modulesPath + "/installer/netboot/netboot-minimal.nix")
  ];

  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEKy/Uk6P2qaDtZJByQ+7i31lqUAw9xMDZ5LFEamIe6l"
  ];

  ## Some useful options for setting up a new system
  services.getty.autologinUser = mkForce "root";

  #<snip>
}

This will take the modulesPath of the nixpkgs instance whose lib.nixosSystem function we are using :slight_smile:

For inputs that aren’t that specific nixpkgs instance, we would need to pass them down through either extraSpecialArgs, which are added to the module arguments, or an overlay, which are added to the pkgs argument.


And that’s it, I think at this point everything should at least have the correct types.

Didn’t think I would have to explain this much when I saw the question, hope it’s not overwhelming. I’m also not sure it will actually work, you’re doing something pretty special.

Feel free to come back and complain if it doesn’t work.

8 Likes

Hi!

I didn’t know that import nixpkgs ... is considered bad (e.g., because of disabling the caching feature). Now, how can I avoid this import when I need the pkgs set, but I want to have some overlays? At the moment, I am doing, for example:

  pkgs = import nixpkgs {
    inherit system;
    overlays = user-overlays ++ host-specific-overlays;
  };

And I know, one can use

      pkgs = nixpkgs.legacyPackages.${system};

to avoid the import. But how can I use the second form with overlays?

Sorry for the diverging question, I can also open another thread, if you want.

EDIT: I am trying to answer my own question:

let nixpkgsWithOverlays = nixpkgs { overlays = my-overlays }; in ...

Would that work?

Same way as with config, edit the nixpkgs.overlays option in your modules:

modules = [
  ./configuration.nix
  ({...}: {
    nixpkgs.config.allowUnfree = true;
    nixpkgs.overlays = user-overlays ++ host-specific-overlays;
  })
];

Or you can pass the overlays into your modules using extraSpecialArgs and use them there. Personally, this is usually why I end up with an inline module in flake.nix.

This only works for nixosConfigurations, unfortunately, for most other outputs you will need to use import after all (or do something with the overlays output).

Yes, I need the pkgs for a separate Home Manager configuration, so I can not use the Nixpkgs module system. And I really want to keep system and user separate… I think extraSpecialArgs is the only way to go.

home-manager can also change the overlays for your home manager config: https://nix-community.github.io/home-manager/options.html#opt-nixpkgs.overlays

And it also has flakes support: https://nix-community.github.io/home-manager/index.html#sec-flakes-standalone

That said, import isn’t totally awful. Using it in “leaf” flakes is totally fine, especially if it would be too hard to get overlays otherwise.

1 Like

Yes, I am using the flakes setup of Home Manager. So it would be

let pkgs = nixpkgs.legacyPackages.${system}; in
 home-manager.lib.homeManagerConfiguration {
   inherit pkgs;
   modules = {{Snippet you provide above which includes overlays}}
}
2 Likes

Hello!! I know how to start! I would like to say thank you for taking the time! Your answer is so detailed, correct and helpful that I got everything figured out and I understood the full cycle a bit better!

I can even see the vm with NixOS (built with nixos-rebuild build-vm (that I didn’t know existed)) correctly hook to Tailscale!

Anazing, I spent a day trying to figure how to manually put together a QEMU command in the right way and it was already done for me :innocent:

Yes! I am trying to setup an ad-hoc “install netbooting” OS, so you read my question correctly!


I feel like I know this topic a bit better but, I don’t know how to discover solution on my own without having to read 100 blog posts that are not up to date, looking at 20 videos and so on.

How do you know that there is for example a modulesPath avaiable as part of the input? Do you look at the code or I can use the repl or eval in some way to deep dive there.


Another question is about nixos-gererator. I used it in the past but when I search for “NixOS netbooting” I can’t find a solution that points me to nixos-generator and the official Wiki uses the module as we do here.

If I understood it correctly I can replace the output.package.generic

packages.${system}.netboot = nixpkgs.legacyPackages.${system}.symlinkJoin {
  name = "netboot";
  paths = with self.nixosConfigurations.generic.config.system.build.topLevel; [
    netbootRamdisk
    kernel
    netbootIpxeScript
  ];
  preferLocalBuild = true;
};

with something like this (coming from GitHub - nix-community/nixos-generators: Collection of image builders [maintainer=@Lassulus]) but with the right format (can’t find the right one):

outputs = { self, nixpkgs, nixos-generators, ... }: {
    packages.x86_64-linux = {
      vmware = nixos-generators.nixosGenerate {
        system = "x86_64-linux";
        modules = [
          # you can include your own nixos configuration here, i.e.
          # ./configuration.nix
        ];
        format = "vmware";
      };
    };
  };

Thanks a lot!

I know right? NixOS is awesome :smiley:

Oh, yeah, except for that detail… About 3 years of using it, lots of reading and answering questions around here, and reading the nixpkgs source. The nixpkgs source is less hard to read than you’d think, but I don’t think there’s a much better way currently.

The details behind modulesPath are documented here, but good luck finding it if you don’t know about it already: _module.args

nix repl can be useful, try it and then use :lf, tab completing afterwards will give you access to everything defined in your flake, including its inputs. But it won’t give you too much information on the nix module system, which is what you would need for this.

On the up side, this is a very well known problem, people like @fricklerhandwerk are currently putting their heart and soul into fixing it. https://nix.dev/ is growing rapidly. But at some level there will always be some friction when learning a new ecosystem, especially when it comes to experimental parts of it :wink:

Yeah, that’s the question. I’m not actually sure how netboot works, I think given the documentation:

vm only used as a qemu-kvm runner
vm-bootloader same as vm, but uses a real bootloader instead of netbooting

You might want vm? No reason to switch away from something that works, though.

2 Likes

@gianarb unfortunately often the answer is „you just have to know, somehow“.

There are countless moving parts in the Nix ecosystem which are not (well) documented. Experienced users usually don’t notice this friction any more because they learned to help themselves. Clearly this is inefficient because it usually takes many months to get there.

If you have the time and energy to help by contributing what you have learned into official documentation, we will help you get things merged. Please check How to contribute to documentation for details. The contribution process (meta-documentation) also needs feedback by beginners, because there also you usually have to „just know“ may things, and we want to make them explicit.

4 Likes