Adding flake to nixpkgs

sullyj3 · July 19, 2022, 3:10pm

Is it possible to add a flake to nixpkgs? The manual only mentions the default.nix way of doing things.

TLATER · July 19, 2022, 3:35pm

Flakes aren’t really designed to be added to nixpkgs - rather, they’re supposed to both be used in a downstream flake and managed that way.

But that isn’t too important, because flakes aren’t themselves what you’re using, but its contents. If you want to upstream the contents of a flake, you’ll need to take the modules/packages/hydra jobs and move them into nixpkgs with a bit of discretion.

What is it that you’re actually trying to do?

sullyj3 · July 19, 2022, 3:40pm

Thanks! I’m not trying to do anything except understand how things fit together, it’s more of a theoretical question.

So what’s the relationship between flake.nix and default.nix? Might you have a project with a flake.nix that references a default.nix?

More generally, it’s not clear to me to what extent flakes subsume the older functionality. I’m relatively new to nix, and I’ve looked at a couple of flakes tutorials, since they’re billed as the way things will be going forward. I’m trying to figure out whether I need to learn how to use default.nix and how that fits together with flakes. Often learning materials covers the details well, but don’t really give the broader picture.

TLATER · July 19, 2022, 5:15pm

Sure, I can probably summarize that, shout if I end up confusing you

But tl;dr: As it happens, I stumbled across this summary of the topic from @jonringer today, in response to a coworker asking almost the exact same question as you (though more focused around what they should actually be putting in default.nix): Nix - Flakes for out-of-tree code - YouTube

If you’re a video person, might be easier to grok than my writing. On the other hand, I think I focus more on showing what the basic idea behind all of this is. You tell me who did a better job ;p

default.nix is actually much simpler than you’d think. It’s a little like index.js in JS land or __init__.py in python land - it’s the file in a directory that nix will evaluate by default. Nothing more than that.

Traditionally, people would put a derivation (read: package definition) for the repository it is in in there, so you can just build the directory of the repository with nix-build and end up with a packaged version of whatever is in that repo.

You’d also often have a sibling shell.nix file, in which people would put a derivation that defines a development environment, so that you can enter the directory, run nix-shell, and start hacking.

Wonderful workflow, except there are a few problems:

This is purely by convention, you could in theory put any nix expression in a default.nix
- That means we’re either completely missing a way to define things like NixOS modules or we’re overloading the one way in which anything can be defined outside of nixpkgs, or both at the same time, depending on your viewpoint.
default.nix cannot really have dependencies
- You can approximate it through import and fetchurl, but this is hard to maintain
Details such as which architecture you’re compiling for must be determined by nix arguments
- The project providing a default.nix has no way to say stuff like "we don’t support building on aarch64-darwin", and having a separate definition for a different architecture is hard

There were solutions to the above problems, mostly that people just actually used default.nix for whatever they liked (and sometimes even shell.nix!), so things were a bit chaotic. If nix was your average project that’d probably have just been it, but instead we got a nice new design to accommodate these emerging use cases.

Flakes are just a way to organize all these things that is supported by nix and assisted by tooling made available upstream. They provide a way to define inputs, track their versions, and a way to define packages, shells, NixOS modules, and even things like hydra jobsets.

The NixOS wiki covers what they look like quite well: Flakes - NixOS Wiki - the important part for understanding here is the output schema.

Among these attributes are packages and devShells - if you wanted to convert a traditional default.nix-based project, you’d simply set packages.${system}.default = ./default.nix; and devShells.$[system}.default = ./shell.nix;.

Note the ${system} - you’d expand that to whatever system you actually support in practice (if you want to support all the systems, you can use flake-utils.eachSystem for that, which calls your definitions in a loop so you can actually use a ${system} variable).

What that would do is that your “default” “package” is now the derivation that you used to build with nix-build, which you can now build with nix build. And the “default” “devShell” is what you would have invoked with nix-shell, but is now invoked with nix develop.

With flakes you can now also define nixosModules, overlays and even hydraJobs, as well as whatever extensions third party projects want to add. For some of these outputs there are handy subcommands already, some may still need some work in nixpkgs or such for convenient support.

Nix can also be smart about these things and verify that you’re putting the correct types of things in each of them through nix flake check, at least to an extent.

Downstream projects now know exactly what you provide, and can conveniently make use of your project through nix commands. All around it’s a win for organization and builds a way to do more distributed nix projects.

This solves our problems:

There is a schema behind this, and our definitions now have names, instead of just convention
- We can also define multiple things in one repository without significant additional readme contents
We have a dependency management system, much like npm, pip, you name it
The architectures are baked into the schema, for those definitions for which they are relevant

As a bonus, since we no longer depend on the contents of the system channels, but instead on the contents of a flake.lock file, our projects are now also actually reproducible from source. This is a (very significant) benefit of the design, not its primary purpose.

Overall, yes, I think they’re the way forward. For learning, I found that flakes both demand a deeper understanding of what nix actually is (and how NixOS uses it), but also let you see more behind the curtains, so you’ll probably understand things better too. State of a year or two ago I found there to be very little in terms of documentation or teaching though, so flakes might be harder to grasp initially.

sullyj3 · July 21, 2022, 3:10am

This is a really comprehensive explanation, thank you! I was familiar with the concept of a flake’s packages.${system}.default and devShells.${system}.default outputs, so it’s nice to make the connection with default.nix and shell.nix that you can just reference them.
I still don’t quite understand why flakes aren’t designed for nixpkgs. Thinking about it more, does it have something to do with the fact that nixpkgs is supposed to be self contained, so having flake inputs wouldn’t make sense?
Whenever depending on nixpkgs in a flake, you use legacyPackages. I had assumed that this would imply the existence of a non-legacy packages which would be something flakey that would be gradually migrated to. Now, looking at https://github.com/NixOS/nixpkgs/blob/af7d2aaa0d7fae44cdef463538833d536e3def1f/flake.nix, I see that there is no packages attribute, so I guess not.

nrdxp · July 21, 2022, 3:16am

I think this has more to do with the fact that flakes are still gated behind an experimental flag, even though in practice they are quite stable. Eventually there may be a plan to add flake integration into nixpkgs, but I don’t think anyone has formulated a solid plan for this just yet. There is also the possibility that flakes eventually start to replace nixpkgs, though there has been a lot of outcry against this in the past.

As far as being no packages, it’s because this output expects a flat package space a single attribute deep, which nixpkgs is definitely not. What we need is a plan for nixpkgs once flakes do eventually stabalize.

Do we try to implement the expected flat package space? Do we start re-exporting external flakes from nixpkgs through the flake inputs? Do we discourage adding new packages to nixpkgs and instead focus on extending the official flake registry for new packages? There are still a lot of unanswered question, indeed probably something that would need to be decided in an RFC, which is why there has been no action taken just yet.

sullyj3 · July 21, 2022, 3:18am

That all makes sense, thanks.

drupol · July 21, 2022, 6:26am

@TLATER I think you should add this in the wiki, these are really valuable information. Thanks !