Is it possible to add a flake to nixpkgs? The manual only mentions the default.nix way of doing things.
Flakes aren’t really designed to be added to nixpkgs - rather, they’re supposed to both be used in a downstream flake and managed that way.
But that isn’t too important, because flakes aren’t themselves what you’re using, but its contents. If you want to upstream the contents of a flake, you’ll need to take the modules/packages/hydra jobs and move them into nixpkgs with a bit of discretion.
What is it that you’re actually trying to do?
2 Likes
Thanks! I’m not trying to do anything except understand how things fit together, it’s more of a theoretical question.
So what’s the relationship between flake.nix and default.nix? Might you have a project with a flake.nix that references a default.nix?
More generally, it’s not clear to me to what extent flakes subsume the older functionality. I’m relatively new to nix, and I’ve looked at a couple of flakes tutorials, since they’re billed as the way things will be going forward. I’m trying to figure out whether I need to learn how to use default.nix and how that fits together with flakes. Often learning materials covers the details well, but don’t really give the broader picture.
1 Like
Sure, I can probably summarize that, shout if I end up confusing you 
But tl;dr: As it happens, I stumbled across this summary of the topic from @jonringer today, in response to a coworker asking almost the exact same question as you (though more focused around what they should actually be putting in default.nix): https://www.youtube.com/watch?v=90P-Ml1318U
If you’re a video person, might be easier to grok than my writing. On the other hand, I think I focus more on showing what the basic idea behind all of this is. You tell me who did a better job ;p
default.nix is actually much simpler than you’d think. It’s a little like index.js in JS land or __init__.py in python land - it’s the file in a directory that nix will evaluate by default. Nothing more than that.
Traditionally, people would put a derivation (read: package definition) for the repository it is in in there, so you can just build the directory of the repository with nix-build and end up with a packaged version of whatever is in that repo.
You’d also often have a sibling shell.nix file, in which people would put a derivation that defines a development environment, so that you can enter the directory, run nix-shell, and start hacking.
Wonderful workflow, except there are a few problems:
- This is purely by convention, you could in theory put any nix expression in a
default.nix- That means we’re either completely missing a way to define things like NixOS modules or we’re overloading the one way in which anything can be defined outside of nixpkgs, or both at the same time, depending on your viewpoint.
-
default.nixcannot really have dependencies- You can approximate it through
importandfetchurl, but this is hard to maintain
- You can approximate it through
- Details such as which architecture you’re compiling for must be determined by nix arguments
- The project providing a
default.nixhas no way to say stuff like “we don’t support building onaarch64-darwin”, and having a separate definition for a different architecture is hard
- The project providing a
There were solutions to the above problems, mostly that people just actually used default.nix for whatever they liked (and sometimes even shell.nix!), so things were a bit chaotic. If nix was your average project that’d probably have just been it, but instead we got a nice new design to accommodate these emerging use cases.
Flakes are just a way to organize all these things that is supported by nix and assisted by tooling made available upstream. They provide a way to define inputs, track their versions, and a way to define packages, shells, NixOS modules, and even things like hydra jobsets.
The NixOS wiki covers what they look like quite well: Flakes - NixOS Wiki - the important part for understanding here is the output schema.
Among these attributes are packages and devShells - if you wanted to convert a traditional default.nix-based project, you’d simply set packages.${system}.default = ./default.nix; and devShells.$[system}.default = ./shell.nix;.
Note the ${system} - you’d expand that to whatever system you actually support in practice (if you want to support all the systems, you can use flake-utils.eachSystem for that, which calls your definitions in a loop so you can actually use a ${system} variable).
What that would do is that your “default” “package” is now the derivation that you used to build with nix-build, which you can now build with nix build. And the “default” “devShell” is what you would have invoked with nix-shell, but is now invoked with nix develop.
With flakes you can now also define nixosModules, overlays and even hydraJobs, as well as whatever extensions third party projects want to add. For some of these outputs there are handy subcommands already, some may still need some work in nixpkgs or such for convenient support.
Nix can also be smart about these things and verify that you’re putting the correct types of things in each of them through nix flake check, at least to an extent.
Downstream projects now know exactly what you provide, and can conveniently make use of your project through nix commands. All around it’s a win for organization and builds a way to do more distributed nix projects.
This solves our problems:
- There is a schema behind this, and our definitions now have names, instead of just convention
- We can also define multiple things in one repository without significant additional readme contents
- We have a dependency management system, much like
npm,pip, you name it - The architectures are baked into the schema, for those definitions for which they are relevant
As a bonus, since we no longer depend on the contents of the system channels, but instead on the contents of a flake.lock file, our projects are now also actually reproducible from source. This is a (very significant) benefit of the design, not its primary purpose.
Overall, yes, I think they’re the way forward. For learning, I found that flakes both demand a deeper understanding of what nix actually is (and how NixOS uses it), but also let you see more behind the curtains, so you’ll probably understand things better too. State of a year or two ago I found there to be very little in terms of documentation or teaching though, so flakes might be harder to grasp initially.
13 Likes
This is a really comprehensive explanation, thank you! I was familiar with the concept of a flake’s packages.${system}.default and devShells.${system}.default outputs, so it’s nice to make the connection with default.nix and shell.nix that you can just reference them.
I still don’t quite understand why flakes aren’t designed for nixpkgs. Thinking about it more, does it have something to do with the fact that nixpkgs is supposed to be self contained, so having flake inputs wouldn’t make sense?
Whenever depending on nixpkgs in a flake, you use legacyPackages. I had assumed that this would imply the existence of a non-legacy packages which would be something flakey that would be gradually migrated to. Now, looking at https://github.com/NixOS/nixpkgs/blob/af7d2aaa0d7fae44cdef463538833d536e3def1f/flake.nix, I see that there is no packages attribute, so I guess not.
1 Like
I think this has more to do with the fact that flakes are still gated behind an experimental flag, even though in practice they are quite stable. Eventually there may be a plan to add flake integration into nixpkgs, but I don’t think anyone has formulated a solid plan for this just yet. There is also the possibility that flakes eventually start to replace nixpkgs, though there has been a lot of outcry against this in the past.
As far as being no packages, it’s because this output expects a flat package space a single attribute deep, which nixpkgs is definitely not. What we need is a plan for nixpkgs once flakes do eventually stabalize.
Do we try to implement the expected flat package space? Do we start re-exporting external flakes from nixpkgs through the flake inputs? Do we discourage adding new packages to nixpkgs and instead focus on extending the official flake registry for new packages? There are still a lot of unanswered question, indeed probably something that would need to be decided in an RFC, which is why there has been no action taken just yet.
4 Likes
That all makes sense, thanks.
1 Like
@TLATER I think you should add this in the wiki, these are really valuable information. Thanks !
1 Like
TL;DR: package discovery and usage goes via nixpkgs, therfor putting flakes there should be easy
late to the party, but i really think this is an important topic.
Many people still only use nixpkgs because discovery and usage of packages outside of it is rather hard in comparison: e.g. system.packages = with pkgs; [ pingus ] is WAY simpler than activating flake support, adding the additional import and then adding inputs.pingus.packages.${pkgs.currentSystem}.default to your systemPackages if you even find your flake (as there is no singular place to go to like nixos search for nixpkgs)
For that reason package maintainers also add their package (e.g pingus and prismlaucher) to nixpkgs, because, again, thats how packages are discovered and used even if they have first party flake support.
Their nixpkgs derivation is pretty much a copy of the logic from their flake files, which i think is bad (as there are now multiple sources of truth) and there should be something like pingus = (importFlake "github:pingus/pingus").packages.default possible in nixpkgs.
I also recently added first party nix flake support for rquickshare and found that rquickshare already is in nixpkgs as well (but only wraps a prebuilt binary). Now i want to replace the binary-wrapper in nixpkgs with the actual build process as it is defined in the new flake (not only to support multiple platforms, but because i think thats how source-based package managers like nix are designed to work?). See issue. Should i go the pingus/prismlauncher route and just duplicate my logic over to nixpkgs? (I’d rather not, but I also dont want to leave it as-is)
I appreciate the feeling from the perspective of someone working on an upstream project, DRY is the imperative after all.
However, from the perspective of a distribution maintainer things change a bit. In practice, NixOS isn’t doing as well as it should, but end users expect to be able to reasonably trust that distro packages are safe to download, that they don’t needlessly duplicate loads of dependencies, that updates happen with proper release cadence in mind, and that they integrate well with other packages.
To achieve all that, unfortunately the nixpkgs package and an upstream build expression must not be tightly coupled. You don’t want to be in a situation where you need to change your commit history on main just because of a nixpkgs-internal breaking change to mkDerivation, for example, and you probably don’t want to be forced to have release branches to accomodate such changes just because you want your package published on nixpkgs either.
Even if you were ok with morphing your project to fit nixpkgs’ needs, delegating repo ownership to third parties with no control by the nix contributors also isn’t really workable. Downstream needs to be able to fix your package if something out of your control breaks, e.g. because of a dependency update.
Maybe you could have your nixpkgs package be the source of truth for your upstream, but then you lose the ability to change anything about your build scripts. This also isn’t really workable.
Flakes aren’t intended to replace nixpkgs. They are tooling to permit more flexible use of nix without being tied to a distro. Flakes’ design fundamentally doesn’t really support the type of project a distro tries to be.
One day we might see a good registry-style project pop up where you can advertise your project’s inherent nix support, but this is very different from a Linux distribution.
For now, I’d go with the pingus/prismlauncher approach. Honestly, NixOS is a bit weird here, other distros have an explicit policy that package maintainers should not be the same people who maintain the upstream, which helps keep this conflict at bay. Also makes it a lil’ harder to sneak in malicious software. You’ll have to make sure to wear the right hat when working on each side.
3 Likes
Definitely not. Using the upstream package makes little sense as the nixpkgs instance will never match (imagine a new nixpkgs instance for every package, that sounds hellish), and creates a security issue.
What you really want is pkgs.callPackage "${builtins.getFlake "github:foo/bar"}/path/to/whatever.nix" but you still have the security issue AND nixpkgs can’t even guarantee the file is callPackage-able AND the flake reference isn’t hashed AND you have the issues @TLATER mentioned.
You could solve the flake reference issue with a fetcher, but builtin fetchers are unperformant (hence banned in nixpkgs) and nixpkgs fetchers would create an IFD here. And adding a flake input would cause insane lockfile explosion for users of nixpkgs… again a perf issue.
Anway, nixpkgs is meant to be standalone nix code, not nix code that breaks on any upstream change.
1 Like
Hm, I guess if you want that, you can change your flakes’ package to nixpkgs.legacyPackages.<package>.overrideAttrs (_: { src = ./.; });, which would be ok of you’re confident your build system only changes very rarely.
Still will get messy over time for all the reasons I already mentioned. Don’t blame me if this breaks 
src = ./. is discouraged:
https://nix.dev/guides/best-practices#reproducible-source-paths
EDIT: Sorry, I know that wasn’t the point hehe.
1 Like
thanks for the great detailed explanation, really appreciate that!
And as i’ve never been a distro maintainer i didnt really think about these reasons. But it makes sense, i’m convinced simply forwarding the flake cannot be the solution 
so another registry like the NUR would probably be a more sensible place to allow third party package discovery via upstream flakes?
and the only way to have a package added to nixpkgs is to duplicate the logic as nixpkgs should somewhat stay the stable integrated core with no outside dependencies, makes sense as well
but couldnt enforcing giving a commit hash to the proposed importFlake in nixpkgs (e.g. (importFlake "github:pingus/pingus/blasblalongsha").packages.default) somewhat solve this issue (security shouldnt be an issue as creating different commits with the same sha is -lets call it- fairly impossible) and therefor all the build instructions etc are also pinned and if something changes a new PR with a new ref has to be made to nixpkgs so the nixpkgs maintainers can look at the flakes source to make sure nothing dubious is going on
the importFlake could also ensure that inputs only allows nixpkgs (so no other flakes) and override that to the calling nixpkgs
or am i missing something else of importance?
but then again, i agree that making the flake nixpkgs compatible (like omitting any IFD etc etc) is probably hard enough to not make it attractive for most upstream maintainers to keep that in mind
The NUR is made obsolete by flakes, but you’re welcome to use it regardless.
I don’t think most people use the NUR as a discovery mechanism.
Your proposed function would be a wrapper around the concepts I mentioned above, I don’t see how it solves the security issue or perf issues (or even callPackage-ability issue).
And we’ve tried basing package definitions off overrides of other packages in nixpkgs, it doesn’t work well and becomes unmaintainable quickly. It’s okay to repeat code.
GH (where most projects live) already blocks colliding hashes. NAR hashes for FODs (or any output-addressed derivation) aren’t about security, they’re about caching the thing so you don’t have to rebuild it and can just use what’s in the store.
even better
yeah ik, i just wanted to put forward that you could omit the flake locking part (as this apperently brings perf issues) by locking the upstream content containing both the nix code to make the package derivation as well as the source by forcing a ref sha. This makes it reproducible as well and the outputs can be cached the same way normal nixpkgs derivations are.
The flake essentially gets repurposed as something similar to a normal default.nix but with constraints on how outputs are structured.
nixpkgs maintainers can then invoke this flake from inside nixpkgs, iff the flake conforms to the aforementioned constraints, and make sure that it (the flake from the upstream repo and its source contents, pinned to that specific commit) produces a usable result although a possibly different version of nixpkgs is used by making some overrides if required.
well, many (if not all) packages in nixpkgs get their src from somewhere (often GH), define a derivation that takes that code and builds it.
All i am proposing is that, if some upstream src already has nix code available that defines the necessary derivation as it uses that for its own build, why not also reuse that?
I dont really get the big difference between having the nix code inside the nixpkgs repo and only the source outside and having both, the source and the nix code, somewhere else. I see that this would allow upstream maintainers to change the nix code without nixpkgs maintainers approving, which I agree is BAD, but enforcing locking to a specific commit omits this issue, right?
If a package is added/updated that way, the reviewing nixpkgs maintainer can check the upstream commit and see if they approve the code. This makes change requests and reviews way harder, sure, but it doesnt seem at all impossible. And making sure this commit builds from nixpkgs has ofc be checked by the PR opener (which in this case probably often is an upstream maintainer).
I simply want to enforce importFlake to be a FOD as well, without requiring the flake locking mechanism
Arbitrary sandboxed code vs arbitrary code.
this indeed is a big difference 
good point
the reviewer would still have to review the nix code, but its upstream, tho pinned and therefor unchangable, now
This helps with the security burden, but not the integration one. Your flake will depend on a nixpkgs that has different inputs. If we use follows, there’s no guarantee your package builds (let alone integrates) without patching. Once something breaks we start having to do complex overrides… And we’re back to having two (or more, for stable/unstable) sources of truth.
And this is before we consider all the advantages of monorepos for projects like this. People already complain every time a tiny detail in the internal libs changes, imagine if literally every package was out of nixpkgs control and needed manual commits to like 10k repositories every time an API subtly changes (and convincing all upstreams that, yes, reducing the maintenance burden is actually worth the occasional sweeping change). I could see this working if all nixpkgs splinter repos were under nix community control, but third party communities definitely shouldn’t be in the graph.
And on top of all of that, flakes are still far from a stable nix feature. There are still frequent calls for just doing away with them altogether. Even if I thought it was a good idea to begin with, I don’t think we should be basing nixpkgs on them anytime soon.
3 Likes