Yes, and in those cases, I assume you would hard-code it instead of using the nixpkgs passed by the “caller flake”.
Honestly, I do think it could have been fixed “in the traditional way”.
If the Nix file is inside a repository, you could have prevented accessing files outside the repository,
and otherwise, you could have prevented accessing files in directories above the root file you evaluate from.
But this wouldn’t be that different from what a flake currently is, of course besides the inputs and schema stuff.
Flakes are still experimental, and we need to find a way forward that solves this, and I think this sort of in-between feature is more amenable to making unexperimental and stable.
There is already an object we commonly use that takes a package set as an input in a very flexible way, overlays. They are already intended to be used by a consumer without regard for the Nixpkgs of the upstream flake. Notice that overlays don’t hard-code a system either. Overlays also shouldn’t depend on any inputs, just on final/prev, making them quite independent. But they are complicated.
The first few flakes we saw were awkward, and then we had a proliferation of flake libraries intended to hide the boilerplate, and “system” and then add niceties. These have had a chance to explore the design space. I think it comes down to two difficult to grok concepts, overlays and callPackage.
If everyone implemented their flakes with proper overlays, well partitioned and organzied, and if everyone including beginers understood overlays, we might be in a better position. Instead we end up with packages that are hard to re-use elsewhere, hard-coding them into modules, and so forth. People are even forced into this situation because it is easier than the alternative. It is also the case the overlays are just inherently complicated, and useful, and hard to learn, and have too many footguns. And we made the correct behavior require expertise, but the easy behavor damaging in the long run.
In some sense, the problem of too many Nixpkgs means that there was also a need, and that people are leveraging this capability, but without the tooling or the idioms to guide them.
I dont think merely having people use niv or raw getFlake is the answer. I suspect we would end up in the same position, with yet another form of having multiple nixpkgs, but now less structured. At least flake inputs are all in one place, localizing the problem and allowing for tooling to handle them. I do want to eventually get back to a proposal from last year about updating how lockfiles treat transitive locks (Reuse input lock files · Issue #7730 · NixOS/nix · GitHub).
One approach I’ve been enjoying is twofold, based around the notion of a “recipe”, a function that produces a derivation (pkg-fun, the default.nix 's you find all over Nixpkgs, a callPackage-able, etc):
-
Remove the need to understand super/self/final and callPackage in 95% of cases. Most people just want to add a leaf package or do a simple override. Not some deep compiler replacement or cross-compilation. Let people define an attribute set/tree, where each value is a recipe. There is an fairly mechanical translation of this form into overlays that adds callPackage and does the right fixed-point magic.
-
Re-use. These more limited forms are easier to handle than overlays, less footguns and less power. Attrsets of recipes can be manipulated predictably and composed together without needing to understand fixed points. They can be grabbed from a flake without reading the upstream lockfile. One can enforce this by a “flake = false” (perhaps that becomes the default?) and passing dummy values as inputs, if it errors it means someone’s overlay accessed a package value directly instead of doing proper composition.
None of this requries any technical change. There is work that can make it nicer; and tooling + templates + guides to make it understandable. Farid was making a tool to more easily manage follows statements, simply having a better interfaces to visualize and manage inputs might go a long way.
7 Likes
@tomberek I fully agree that the “recipe as a function from derivations to (attribute set of) derivation” pattern has the exact right properties in your framing of learnability and composability, and that we should promote it to beginners. This is pretty much what we’re doing in the tutorials on nix.dev, this is the direction pkgs/by-name is taking, and where an overhaul of the Nixpkgs manual could help to instill and solidify the right ideas in casual users.
But this doesn’t address the problem of organizing remote sources, because it’s a completely orthogonal issue. I’d even go as far as saying that the recipe pattern is at odds with the “everything is a flake” pattern, because at the point where you package everything as a function at the level of default.nix, you may as well provide default values for your inputs based on your local pins, however you manage them, and leave the rest to callPackage (one of those great misnomers) and override. Almost inevitably you’ll then have to ground everything in some version of Nixpkgs, ideally controlled by the consumer.
All that remains is a user interface for the imperative (essentially end-user) use case of the likes of nix run, which, as I argued multiple times, is probably better suited as a Nixpkgs idiom since it would much better live off derivation metadata rather than some superimposed rule for what an “installable” or “app” is. All Nix itself would need to provide is the language for composition and process isolation for realising derivations.
4 Likes
Exactly. Overlays and callPackage are the secret sauce of nixpkgs. You can try to make them easier to use, but if you replace them with something less powerful the end result will be much worse than nixpkgs.
This is what infuse.nix does.
I’ve been working on it for the last nine months, and will be releasing it next week have released it (and, shortly afterward, two other projects that use it). I’ve converted my (massive) set of overlays to use infuse and can’t imagine ever going back.
Here’s what my override of pkgs.xrdp looks like today; notice that this requires a nested overrideAttrs:
infuse pkgs {
xrdp.__input.systemd.__assign = null;
xrdp.__output.env.NIX_CFLAGS_COMPILE.__append = " -w";
xrdp.__output.passthru.xorgxrdp.__output.configureFlags.__append = ["--without-fuse"];
};
Here’s what it looks like without infuse:
pkgs // {
xrdp = (pkgs.xrdp.override {
systemd = null;
})
.overrideAttrs(previousAttrs: {
env = previousAttrs.env or {} // {
NIX_CFLAGS_COMPILE =
(previousAttrs.env.NIX_CFLAGS_COMPILE or "")
+ " -w";
};
passthru = previousAttrs.passthru or {} // {
xorgxrdp = previousAttrs.passthru.xorgxrdp
.overrideAttrs (previousAttrs: {
configureFlags = (previousAttrs.configureFlags or []) ++ [
"--without-fuse"
];
});
};
});
}
We always had the right power tools, it just took a while to figure out how to make them less awkward.
6 Likes
I have some qualms with overlays and callPackage, e.g. that input packages are derived implicitly from the argument list, which creates problems when you don’t have a single global namespace, as happens with e.g. Haskell packages. Then you have the issue that you want to depend on some system package called x, but there’s a Haskell package with the same name, so you can’t depend on it! Horrible! Of course you could have some encoding system where you can prefix the name with _root_ such that it’d _root_x to get the top-level x, and _root__haskellPackages_x to get the one in the haskellPackages namespace, but this is also horrible.
callPackage in its current form is not a viable option.
Other than that, I very much endorse the direction, but especially in this direction it does not make sense to use flake inputs as we use them now.
I feel like a good first step would be to give non-flake Nix code the same purity guarantees that flakes get, i.e. as I described above, for any code in a Nix repo, restrict access to files checked in to Git, by default. It would break very little code I imagine. Doing it for files outside of a Git repository would break much more code.
2 Likes
OTOH, perhaps the more important part,
as an end-user, you want to be able to go into a repository and just nix build.
But as a library consumer, you probably do just want to do the equivalent of flake = false, and
you’re probably right, it would probably make sense to have this be the default. This would be a big breaking change though.
IMO this is because makeScopeXXXXXX and callPackage functions aren’t powerful enough:
EDIT: Where I was going with the linked snippet is that instead of feeding just a lambda into callPackage, we could say a “recipe” is a structure with a lambda and the metadata explaining how to fill in the lambda’s arguments. In particular, we could try and express the solution to your problem as “take X from the scope two levels up”
It is notable that we’re not discussing flakes anymore, it likely never was about flakes 
1 Like
infuse
@amjoseph Looks interesting! I’ve been using a similar construct, using functions as the delimiter for traversal:
using pkgs {
xrdp = { xrdp }: xrdp.override {pulseaudioSupport = true;};
my-hello = {hello}: hello:
pkg-from-file = ./default.nix;
pkgA-from-dir = ./a;
acmePackages = ./pkgs;
my-other-hello = {hello}: hello.overrideAttrs {name = "my-name";};
my-cmd = {runCommand}: runCommand "mine" {} "touch $out";
python3Packages.thing = {requests}: requests.overrideAttrs {name = "my-requests";};
lib.add1 = {}: a: a + 1;
data.b = {lib}: lib.add1 12;
_hiddenDep = {stdenv, lib}: stdenv.mkDerivation ....;
}
This allows for a simple lib.recursiveUpdate for merging. So I can imagine incorporating infuse as well, to simplify the cases of override/overrideAttrs. In my implementation i started adding some nice features with an “expand” step that would interpret some things to add niceties ( a = {}; roughly means a = {a}: a;, or handling path directories. Your lib is adding another set of handlers specialized to overrideAttrs and override (plus with better names!). I do like your idea of interpreting lists as pipes, so far I’ve only been map’ing through them. Mind if I incorporate some of those ideas/conventions into my lib?
random ideas
Two more random ideas to bounce around:
-
Make use of the otherwise useless
<thing>syntax in flakes to access/define inputs instead of NIX_PATH. Sure, it’s an easy way to do the wrong thing, but easy to syntactically detect without eval, build tooling can manage them, less weaving of self or inputs everywhere. You can get the same behavior as Go, where importing a new module anywhere is detected and becomes part of the root, tracked and locked. -
Subflakes don’t quite work the way people expect, but peolple keep trying to use them to fill a need. Introduce a new concept “crystals” (just a tiny snowflake!). It’s definition looks like another flake, but it can never have its own lockfile. Instead its inputs declarations bubble-up the to host flake’s lock, and all the input values flow back to the crystal during eval. This is similar to the Cargo.toml and workspaces concept. Allows to group and subdivide input definitions (PoC: init: crystal support · flox/nix@8a684bd · GitHub).
Flakes also provide standardization and a URL scheme to give things names. Inputs are also needed in some form, but i agree they should be improved.
callPackage and overlays
IMO this is because
makeScopeXXXXXXandcallPackagefunctions aren’t powerful enough:
@SergeK Can you clarify? I’ve been digging into these two and making variants that are aware of hierarchical attrsets / package sets. Or do you mean something else?
3 Likes
Better late than never, I guess. Two mirrors:
- https://codeberg.org/amjoseph/infuse.nix (prettier markdown rendering)
- https://git.sr.ht/~amjoseph/infuse.nix (send patches here)
Tutorial and the announcement will be posted tomorrow. And a few other things. And I will fix all the typos in the README.md.
I will bump the version number to 1.0 once I double-check everything.
And I will catch up on replies to comments. I’m not ignoring anybody, I’m just backlogged!
This is what my overlays look like using infuse.nix.
The hardest part to get right was how to deal with empty attrsets; this took a really really long time to figure out. It’s very counterintuitive.
3 Likes
Exactly. But there wasn’t much demand to give them this power, mostly because…
… until very recently – hierarchical packagesets were hellaciously painful to override (both .override and .overrideAttrs). This isn’t just about overlays; nixpkgs itself uses .override and .overrideAttrs very extensively.
Making hierarchical packagesets as easy to deal with as flat packagesets was a major motivation for infuse.nix; see this example headline example. Now I think we are ready to consider a version of (or alternative to) callPackage that allows hierarchical names. It’s a shame that nix doesn’t allow hierarchical arguments though, like
{ lib
, stdenv
, haskellPackages.compiler
}:
I’m pretty sure this doesn’t exist yet, but it should. There was a lot of pressure to not create any new hierarchical packagesets (there is no rustPackages) because of how painful overriding was.
You can do this with --option pure-eval true – no flakes or --extra-experimental-features needed. I don’t think the purity comes from flakes, it’s that flakes take away the --option pure-eval false escape hatch.
2 Likes
Is there a way to support this in flakes-compat?
Also, for a dose of crazy, check out this reuse of the <thing> syntax.