This is a bit of a beginner question, but I can’t seem to find a clear answer online.
There is a project foo that is managed with a nix flake. If I run nix build inside the foo directory, it will produce an executable.
I have a project bar that I’m managing with a nix flake. I want to use the executable produced by foo as a buildInput in my devShell in the bar flake. Whats the best way to do this?
My idea is have foo as one of the inputs of my bar flake. Then I think I could put something like this in the buildInputs of my bar devShell: foo.packages.default.${system}
Is this correct? Is this the optimal way to do things? Is it possible to do this through overlays instead? Should I be using defaultPackage instead of packages.default?
Yes, but that’s a bad idea within the context of flake interaction. Overlays are for modifying nixpkgs, and since your setup doesn’t involve nixpkgs at all, you shouldn’t use them for this.
defaultPackage is deprecated, you should not use it.
You can use packages.default, but depending on what foo exactly contains packages.foo may be nicer.
This is opinion, but it’s ultimately a rough edge of the flakes design to which there is no real solution. IMO you shouldn’t work around this, and just use the provided solution, which is the .follows mechanism. I’ve taken to using flint to make sure I do that correctly.
The downside of using .follows is that there is no guarantee your nixpkgs input versions are compatible. You also can’t use community caches if you have more than one input using nixpkgs. Again, there is no perfect solution here, this requires actual design changes within flakes, and is one of the many reasons I avoid them where possible.
Yes of course, and I would argue this is better since your users then won’t have to fiddle with system manually, among other benefits. .follows + using a .packages output will still cause nixpkgs instance proliferation (which bloats memory usage and slows down eval), and overlays won’t (I recall someone actually benchmarking this and finding that a single overlay for a leaf package being basically negligible).
(Of course I would still recommend using .follows if for no other reason than to save the time downloading another nixpkgs tarball, and collapsing the lockfile a bit.)
Ah, interesting. I guess I should look more into overlays. I have seen them used a fair amount which made me wonder if I was going about things the wrong way. But it seems the solution I have is fine for now.
@zimbatm is wrong here.
The only thing .follows will do is ensure the nixpkgs revision is the same.
You are still creating a new nixpkgs instance from the same revision (memoized via nixpkgs.legacyPackages.${system}, but nonetheless) - and you can easily prove this to yourself by creating a flake which exposes an unfree package and trying to use that in your config.
In such a case, the only way to use that package is via an overlay. (Well the upstream could also configure in such a way to allow unfree packages, but that seems borderline malicious, and that will again only apply to their instance.)
EDIT: and worse, many upstreams will actually use import nixpkgs { system = ...; } and such function calls are not memoized in nix. So they are really creating new nixpkgs instances even if they are all using the same system and config.
That doesn’t prove anything since the config of the nixpkgs input cannot be modified at the input level (which, yes, another reason why flakes still fall short almost a decade down the line :D). The equivalent of this test case would be to use nixpkgs-unfree, but that doesn’t really show much.
Obviously you have two nixpkgs instances if you have two nixpkgs instances, and if you instantiate a NixOS configuration it will create its own nixpkgs instance via an un-memoized import; there is no way around this.
However if you don’t involve a NixOS instance, and simply end up with e.g. a package in a devShell, and have a deeply nested tree of flake inputs where all of them use .follows, the memoization should mean that you don’t end up with 1000 instances actually being evaluated.
That’s what I mean by “proliferation” - the duplication on the NixOS configuration level is unfortunate, but not bad enough to call “proliferation” IMO.
Overlays are the way around creating another nixpkgs instance on top of the NixOS one (or whatever code you’re writing that instantiates nixpkgs and possibly configures it).
NixOS is kind of irrelevant here - I should have said “in any configured nixpkgs instance” to be more general. In a devshell or wherever, at the very least you probably have at least two instances - your own instance and the memoized instance of every “conformant” flake (i.e. the flakes that use legacyPackages.${system}) as it’s pretty normal to end up configuring your own nixpkgs instance. On some systems even two nixpkgs instances is a lot. And again, while it would be nice to believe that most upstream flakes are “conformant”, in reality it’s about 50/50 (slightly in favour of the non-memoized version).
I’m also not seeing a strong argument against overlays. They can be complex undoubtedly, but that’s hardly relevant for a new leaf package. It’s not what they were “intended” for, but that goes for basically any code written around flakes and then a lot of the other nix code too - I don’t see that as a reason to not write code that gets the job done well. (Especially since overlays predate flakes and the creator could have hardly imagined the more “creative” uses of overlays.)
What you can do in your root flake is create your own nixpkgs.url = "path:./nixpkgs"; sub-flake, and inject it into all the conforming flakes using the “follows” mechanism. Then you have a single instance of nixpkgs, with overlays for the project overrides. And assuming flakes depending on nixpkgs unfree use GitHub - numtide/nixpkgs-unfree: nixpkgs with the unfree bits enabled as an input, then the whole tree can be collapsed. That also helps catch flakes that might import nixpkgs.
Granted, it’s a bit of a faff to annotate and manage all the follows and inputs. But it has the advantage of being explicit. What’s really missing here is SemVer support for Flakes, so the inputs can be auto-collapsed.
The main issue with overlays is that they are touching a global namespace. For each dependency you add, there is a risk of key collision. And the only way to find out is to read the code of each dependency (which might change on updates too). The end-result can also change based on the ordering of the layers. And based on my personal experience, fixpoints have a higher change of triggering the dreaded infinite recursion issue. This implicit nature, combined with final and prev, can make reasoning really challenging for newcomers, and at scale.
Part of the namespacing issues can be solved trough convention. Similar to what Ruby is doing with their Gems where each gem gets the prefix of their gem name. Nothing prevents Ruby code to do crazy monkeypatching, but the conventions is asking not to do this. We just don’t have that in the Nix ecosystem yet.
It’s not follows vs overlays. I don’t think you read anything I wrote. Again, nothing I said implied overlays are easy to use, just that they are necessary given current design decisions.
follows can also lead to hard-to-debug errors. Yet no amount of follows or nixpkgs-unfree will fix other flakes’ usage of import.
(And recommending subflakes of all things is far more complex and prone to nix bugs than my suggestions )
and then a lot of the other nix code too - I don’t see that as a reason to not write code that gets the job done well. (Especially since overlays predate flakes and the creator could have hardly imagined the more “creative” uses of overlays.)
)