I held my tongue after zero-to-nix, but I feel I have to point out how badly this reminds me of the old Haskell days when Stack was coming onto the scene. A proprietary company didn’t like the status quo, so they came in and, rather than fixing the problems, made a bunch of their own solutions. They were good solutions, but nonetheless they created a massive divide in the community that it still reels from to this day. The old problems still existed, so if you had them, you were told just to use Stack instead. Many prominent members of the Haskell community outright left or at least became drastically less involved as a result of this divide. After a long while, “Nix-style-local-builds” became the default in cabal-install and things have largely settled, but there are still remnants of this divide.
DetSys is risking such a divide. Leaving behind the problems it doesn’t care for in favor of its own solutions makes me extremely worried we’re about to see the same thing happen yet again. People have many legitimate issues with flakes that need to be addressed, and ignoring them in favor of just plowing forward with DetSys’s own ideas isn’t helping.
I’m happy DetSys has a vision forward that it is excited for. That’s a good force to have. But making that vision a reality is putting the cart before the horse if you don’t have community support.
Thanks for building this and making it available for free.
I admire your wish to make the situation with flakes better, even if it means building something around it. But I think the problem with versions can only be solved from within Nix. Here’s why:
With your solution on FlakeHub, you now can depend on packages using semver, and when updating the flake, those dependencies get updated to the highest version allowed by semver. What it does not do (to my knowledge) is updating the dependencies of your dependencies as well. This means, that your dependencies have to update their dependencies very often so that your package can be up to date too.
NPM for example completely ignores the lockfile of dependencies, and also updates dependencies of dependencies, deduplicating as much packages as possible in the process. A dependency is still able to force specific versions of it’s packages using a shrinkwrap file.
For flakes, I think we’ll have to do something similar. By default, only the top-level lock-file should be respected, for everything underneath semver should be the way to go, with the option to opt into respecting the lockfile of a specifc dependency arbitarily deep in the dependency tree.
If we want do avoid extreme package duplication, I don’t see a way around this.
What’s currently missing to do this is a way for nix to know what versions of a flake are available. And we’ll have to agree on a protocol or something like that, so that nix knows how to get this information. As soon as we’ve settled on a protocol, It’d be very cool if FlakeHub made the reference implementation
I have only used Nix for a year, so I started building my entire system around flakes. I have to say, I absolutely love it. Unfortunately all the good resources on Nix (NixOS manual, Nixpkgs Manual, nix.dev) do not focus on Flakes because they are still experimental. This caused a lot of headaches for me, the only good resources I had were 1) asking tons of questions on the Matrix channel, as there are many experienced people who actively help you out, and 2) zero-to-nix, which definitely helped but only gives a very few and short examples on how to effectively work with them.
I know people mention it all the time, but I will say it yet again: It would be absolutely beautiful if a group of experienced people at DeterminateSystems could spend their time on stabilizing flakes.
Having easy, reliable, official and detailed manuals on doing various things with flakes and the new cli is something I am dreaming of.
The documentation situation has definitely gotten better, with unofficial resources such as the flakes book or third party repositories on GitHub, but there is still a ton missing on those documents.
This is my personal opinion, and I know some people might be sick of hearing this because it was mentioned so often, but I do not know how else all of this should be effectively pushed forwards!
Aside: I’m not convinced that this is necessary or correct. We already have the follows mechanism where an input flake’s possibly-stale dependencies can be matched to the version in the … uh… calling environment. Regardless, there will be subtleties that we all need to discover through experience and experiments like this.
You’re right about the protocol, though: how do flakes publish a semver-compatible version number to be found. Obvious options include github (and similar) release tags, a version metadata attribute added to the flake schema, and more.
I’m not sure where flakehub is currently getting that data, and of course the point is that it can change and evolve. But the nice thing that it adds today is (what appears to be) semver match resolution in the http path to the flake, which allows more options and broader matching than current options (like following a branch that might or might not exist upstream).
I think only the root flake lock matters, in terms of pinned revisions, but the named inputs can be changed via follows to point to different sources (ie, branches, typically)
It’s interesting that it doesn’t necessary needs to be a part of the nix tool. A model with “server side” resolver, where the cli only knows about URLs, lockfiles, and redirects that go into the lockfile, and the actual semantics of version resolution is encoded into a server serving a particular URL could be a viable long-term solution.
A prior art here would be Deno, which works much the same way:
$ exa -l
.rw-r--r-- 3 matklad 23 Aug 08:44 deno.json
.rw-r--r-- 63 matklad 23 Aug 08:45 main.ts
$ cat main.ts
import * as colors from "https://deno.land/std/fmt/colors.ts";
$ deno run main.ts
Warning Implicitly using latest version (0.199.0) for https://deno.land/std/fmt/colors.ts
$ cat deno.lock
{
"version": "2",
"remote": {
"https://deno.land/std@0.199.0/fmt/colors.ts": "a7eecffdf3d1d54db890723b303847b6e0a1ab4b528ba6958b8f2e754cf1b3bc"
}
}
(no judgment on which model is actually better, but both are viable)
I also want to point out that this is PR speak and gaslighting, by which I mean telling people their problems are a part of the solutions and how they’ve solved so many of them.
This is a typical apology on everyone else’s behalf; they’re not sorry for what they did, they’re sorry that someone doesn’t like it. It also serves as gaslighted wording that you’re supposed to be excited about this.
This does not address existing concerns at all. Least of all the ones alluded to by the comment @grahamc is replying to.
This is white-knighting, another form of gaslighting, suggesting DetSys must be heroes despite as-of-yet unanswered criticisms.
This is PR speak for “we don’t care that it isn’t going the way we want; we’re doing what we want anyway.”
From a journalistic perspective, I cannot take DetSys’s claims seriously yet. They clearly want to further their own designs, which may well be good ones. But the tactics used here leave a bad taste in my mouth
Have the authors of the flakes currently added to flakehub in any way accepted to be indexed? This will drive more traffic to their flake and increase the maintenance burden. related
I agree with others that pushing for stabilization of flakes and fixing issues that currently exist around them should be prioritized.
This effort, although it might be nice, is like building a castle on sand to me - as long as flakes are not stable and officially the way to go (in my perception it is not), I see no point in building something like flakehub.
Also, as others have notes, this might create more problems (around trust for example) than it solves. Also: A centralized infrastructure for a distributed tool? GitHub is also a centralized infrastructure for a distributed tool and we see all the problems with it… Thus I can only say that I don’t like what’s happening here.
Your workforce - IMHO - would have been needed elsewhere much more dearly!
I think the contentious reception here is more of a function of Determinate System’s general reputation with the Nix community (or at least its segment here) than of the merits of FlakeHub itself.
@grahamc is correct that a comment on RFC 0144 suggested doing this as an external project before doing it as an RFC:
However, since this looks like it could be easily implemented outside of Nix, I’d strongly suggest doing so first so that everyone can judge the usefulness.
So I don’t think it’s fair to turn around and blame DS for not doing this work upstream. (Elsewhere, he said upstreaming it was an eventual goal.) Similarly, while I personally wish more effort were being put into stabilizing flakes before building even more on top of them, grahamc did say that this was a weekend project:
In the meantime, the proof of concept of FlakeHub took about a weekend to put together.
So it seems unfair to dunk on FlakeHub for representing effort not put into some other aspect of Nix or flakes. Let him who has not spent his last weekend stabilizing flakes throw the first stone, or something.
Like many long-running marital spats: it’s not that the dishes were left out, it’s what the dishes being left out represents; it’s not the proximate issue being litigated, it’s every previous unresolved issue being re-litigated with the current issue as a proxy. It’s not that FlakeHub has technical issues outside of what one would expect from a weekend project, it’s that FlakeHub (and the manner in which it was developed and announced) is another instance of a pattern of behavior from DS, and it’s that longer pattern that people are reacting to.
I’m not a marriage counselor or experienced community moderator, so I don’t have any real solutions to offer. Maybe DS should work with Nix leadership to host or sponsor a flake hackathon to implement all the unimplemented RFCs? Maybe make a blog post about which in-progress RFCs you’re shepherding? We’re not going to get along better by having this same fight all over again, that’s for sure.
@Jaculabilis that hackathon with the rfcs and a reward from a sponsor is a great idea, there are many bigger technologies that have done this and it mostly leads to great success!
Is there a plan to release FlakeHub source code in the future?
Having flakes for reproducible and automated builds is good, but if the workflow depends on an proprietary upstream service, this is really concerning.
I’d like to point out that if the Flakes project had been started by most other persons, it would have been a community/third party project instead of something part of code Nix. There would be no “upstream” in the way we are talking about here. And as @Ericson2314 already pointed out, we’d be probably better off that way as a whole. (Of course it is too late now, Flakes have picked up too much momentum too fundamentally change course)
A lot of peoples’ frustration with Flakes comes from the fact that they have been developed in a way which would be more appropriate for an external project to Nix than something weaved into its core functionality. This impedance mismatch has proven difficult to resolve and is an ongoing cause for tension, as clearly visible in this thread.
I can totally understand why people would like to see these issues resolved in upstream Nix instead. But I’d argue that moving some of that development out into external projects first, small experiments which can iterate at a faster pace, may actually be a good thing.
It seems like the source code to flake hub has not been released though. I didn’t see any mention of the intention to do so either, but I might have missed it.
There is certain core capability that had to be in nix itself, and some in nixpkgs to make it a viable experiment, and there is the experimental features flag / marker around it to address the concerns (about dependencies and assumptions sneaking in to the rest of nixpkgs, for example). There’s probably room for some quiet armchair analysis about exactly where the boundary lies or how to improve the pattern/process in future, but we are now clearly at the point where enough is in place to usefully enable such external experiments.
I’m really tired of arguments and implications about people’s intent, especially when it seems like any choice is wrong in someone’s eyes. That’s not a comment about this post in particular, by the way. However, if we keep on like this in general, no experiments can succeed and no material progress will be made.
fh is a CLI for FlakeHub, just as, say, gh is a CLI for GitHub. So no, it doesn’t really work independently from FlakeHub, although some of its functionality, like fh init is more broadly useful.
