Introducing FlakeHub

IMO, it would be really great to have Determinate Systems publicly back https://github.com/NixOS/rfcs/blob/c8569f6719356009204133cd00d92010889ed56d/rfcs/0136-stabilize-incrementally.md

3 Likes

Iā€™m a relative newcomer to the Nix world, but this whole flakes situation is a complete mess. Just for the record, I have no affiliation with DetSys.

From a documentation perspective, from a usability perspective, from an optics perspective. This thread takes what is ostensibly an effort to move things forward and devolves into a slinging match about everyones personal opinions on the merits of flakes rather than FlakeHub itself. Can you see how this is a bad look for the projects approachability?

Iā€™m too new to have a strong opinion either way on flakes or FlakeHub specifically but surely if there were no problems with flakes then FH wouldnā€™t have been created!

As a n00b being able to track my dependencies without having to keep track of upstream releases seems like a hugely beneficial thing. And DetSys has done nothing to earn the skepticism being slung around in this thread have they (please enlighten me!)?

I guess this post is a polite way of saying youā€™re your own worst enemies at this point! Youā€™re riding the crest of a hype wave as a project atm and seeing real adoption, real investment and real new users (hello, I am one). That will all evaporate if the pervading attitudes on display here donā€™t change imho.

24 Likes

On one hand, there is a community who loves Nix, want to make it great and open source, slowly but surely. On the other hand, you have companies willing to make profit (thatā€™s what they exist for obviously) with Nix by making MVPs (minimum viable product) based on Nix as much/fast as possible.

At some point, itā€™s hard to stand in both camps IMO.

15 Likes

I just love binary evil/good oversimplifications:

  • Clearly DetSys folks arenā€™t a part of the community who loves Nix, and edolstra/grahamc havenā€™t been pillars of said community for years.
  • Clearly the community is unanimous on wanting flakes to stay experimental, and there isnā€™t a sizeable chunk of the community that has been relying on said experimental feature for years. Everyone just wants things to continue going slowly but surely, because itā€™s clearly working!
  • Clearly profit and a companyā€™s success canā€™t be aligned with making Nix great and open source.

I could keep goingā€¦ Of course itā€™s going to be ā€œhard to stand in both campsā€ when those camps are made up strawmen.

38 Likes

Is there a resource that can explain to a non-programmer new user what the hell a flake even is/does? Iā€™ve attempted to read multiple explanations making what are probably correct statements, that in practice leave me coming away with no clear understanding other than they go further as an abstraction, supposedly fixing some important problems while also adding important problems and being the future.

Things like Flakes - NixOS Wiki are impenetrable for gaining a grasp on them, if you donā€™t already know why they are needed.

Iā€™m pretty sure Iā€™m ultimately going to need pictures or flowcharts of how all these pieces relates to things, or something that walks through the syntax of a comprehensive real example.

5 Likes

Sure, there is some useful info here: Flakes ā€” nix.dev documentation

4 Likes

I need to put some work into this page but this may help unlock things a bit: Nix flakes.

5 Likes

Thatā€™s definitely one of the best Iā€™ve seen so far, thanks.

Iā€™m reading it as ā€œitā€™s a folder that contains a project that generates some (hopefully) reliable output of nix expressions, used for any and all possible system operationsā€

Which now makes sense why github is seemingly involved/mandatory

2 Likes

To be clear, GitHub is not necessary for using flakes. Flakes can be in your local filesystem, they can be on a non-GitHub Git provider, they can be tarballs fetched from a URL, and so on. They just have to be a directory structure with a flake.nix at the root with a set of outputs that others can use.

FlakeHub currently only supports publishing flakes from GitHub because GitHub Actions provides a built-in authentication story (courtesy of JSON Web Tokens), but we hope to support other providers down the road. Nix, however, has no such dependency.

8 Likes

Did I miss the discussion that this could be part of nix itself? It seems like there is a need for such feature.


After browsing a bit around and reading the usage policy I am missing a sense of responsibility and quality for the content promoted on the site. Coincidentally I had recently a very lengthy discussion about a theoretically platform named flakehub and the majority of people agreed that it would require some sort of quality control to not be a loaded gun pointed at your foot.

7 Likes

You might be looking for [RFC 0144] Versioned Flake References by figsoda Ā· Pull Request #144 Ā· NixOS/rfcs Ā· GitHub

6 Likes

Donā€™t think the accusations of ill-will help here, but Iā€™m also a bit concerned about this. If proper support for a feature like this lands in nix, will FlakeHub be retrofitted somehow?

Given that it somewhat abuses input urls (which seems to be the main criticism of the project), I figure that would also cause a fair bit of breakage, potentially recursivelyā€¦ Flakes arenā€™t stableā„¢, but this still seems like a very relevant question given how user-oriented this website is. Glossy website + catchy naming and app-store-like functionality feels like it will be a newbie magnet. Is FlakeHub unstable too?

Having an index into the developing flake ecosystem is cool, though, and this definitely highlights a clear need for a new nix feature. Even the operators seem nice and thought out. Iā€™d start using it immediately if it actually made use of the flake input schema.

18 Likes

For sure, and easily! We would like this behavior to be accepted into Nix. On the RFC for version matching behavior, a core Nix dev said the right way to do this is to do it outside of Nix to prove the utility. This is step one, and there is no reason it should cause breakage :).

FlakeHub is stable. Weā€™ve used it internally for several months to great effect.

Iā€™m glad you think so! It does use the Flake input schema and features that are Native to Nix. Weā€™re not breaking the Nix or flake model here, so you should be good to go if youā€™d like to :+1:.

12 Likes

Today was the first day of bioinformatics class. A pretty big part of it was:

donā€™t worry, weā€™ll stuggle with conda together and master it eventually

And oh man, itā€™s gonna be a struggle. Weā€™re going to be dealing with ā€œI donā€™t remember which commands I ran but now it doesnā€™t workā€ for weeks.

Iā€™ll be making a flake for my own use and eventually sharing it with the class: ā€œall the tools you need for CHEM 4731 in a single devshellā€. I was going to distribute it as a link to the github repo, maybe Iā€™ll use flakehub instead.

With the exception of ā€œHow do I install Nix?ā€ the docs seem more about convincing an informed skeptic to try it out. It might be worth adding a section that guides a newbie along the shortest path from ā€œallegedly this flake thingy will give me a shell where I can run bowtie2ā€ to a shell where they can run bowtie2.

3 Likes

Totally understood, no worries. Iā€™d like to see the primitives built in to Nix, too.

What steps are being made to that aim right now?

1 Like

Oh ffs really? Ok so
ā€¦
6/10 need to use but donā€™t want to reward bad behaviour.

0/10, this is bad behaviour. These are blatantly toxic comments, using even positive elements to make personal criticisms.

Iā€™d encourage everyone to keep their criticisms constructive and focused on the work, but this is entirely over the line, is unacceptable, and should be removed.

29 Likes

I held my tongue after zero-to-nix, but I feel I have to point out how badly this reminds me of the old Haskell days when Stack was coming onto the scene. A proprietary company didnā€™t like the status quo, so they came in and, rather than fixing the problems, made a bunch of their own solutions. They were good solutions, but nonetheless they created a massive divide in the community that it still reels from to this day. The old problems still existed, so if you had them, you were told just to use Stack instead. Many prominent members of the Haskell community outright left or at least became drastically less involved as a result of this divide. After a long while, ā€œNix-style-local-buildsā€ became the default in cabal-install and things have largely settled, but there are still remnants of this divide.

DetSys is risking such a divide. Leaving behind the problems it doesnā€™t care for in favor of its own solutions makes me extremely worried weā€™re about to see the same thing happen yet again. People have many legitimate issues with flakes that need to be addressed, and ignoring them in favor of just plowing forward with DetSysā€™s own ideas isnā€™t helping.

Iā€™m happy DetSys has a vision forward that it is excited for. Thatā€™s a good force to have. But making that vision a reality is putting the cart before the horse if you donā€™t have community support.

58 Likes

Thanks for building this and making it available for free.

I admire your wish to make the situation with flakes better, even if it means building something around it. But I think the problem with versions can only be solved from within Nix. Hereā€™s why:

With your solution on FlakeHub, you now can depend on packages using semver, and when updating the flake, those dependencies get updated to the highest version allowed by semver. What it does not do (to my knowledge) is updating the dependencies of your dependencies as well. This means, that your dependencies have to update their dependencies very often so that your package can be up to date too.

NPM for example completely ignores the lockfile of dependencies, and also updates dependencies of dependencies, deduplicating as much packages as possible in the process. A dependency is still able to force specific versions of itā€™s packages using a shrinkwrap file.

For flakes, I think weā€™ll have to do something similar. By default, only the top-level lock-file should be respected, for everything underneath semver should be the way to go, with the option to opt into respecting the lockfile of a specifc dependency arbitarily deep in the dependency tree.

If we want do avoid extreme package duplication, I donā€™t see a way around this.

Whatā€™s currently missing to do this is a way for nix to know what versions of a flake are available. And weā€™ll have to agree on a protocol or something like that, so that nix knows how to get this information. As soon as weā€™ve settled on a protocol, Itā€™d be very cool if FlakeHub made the reference implementation :slight_smile:

4 Likes

I have only used Nix for a year, so I started building my entire system around flakes. I have to say, I absolutely love it. Unfortunately all the good resources on Nix (NixOS manual, Nixpkgs Manual, nix.dev) do not focus on Flakes because they are still experimental. This caused a lot of headaches for me, the only good resources I had were 1) asking tons of questions on the Matrix channel, as there are many experienced people who actively help you out, and 2) zero-to-nix, which definitely helped but only gives a very few and short examples on how to effectively work with them.

I know people mention it all the time, but I will say it yet again: It would be absolutely beautiful if a group of experienced people at DeterminateSystems could spend their time on stabilizing flakes.

Having easy, reliable, official and detailed manuals on doing various things with flakes and the new cli is something I am dreaming of.

The documentation situation has definitely gotten better, with unofficial resources such as the flakes book or third party repositories on GitHub, but there is still a ton missing on those documents.

This is my personal opinion, and I know some people might be sick of hearing this because it was mentioned so often, but I do not know how else all of this should be effectively pushed forwards!

14 Likes

Aside: Iā€™m not convinced that this is necessary or correct. We already have the follows mechanism where an input flakeā€™s possibly-stale dependencies can be matched to the version in the ā€¦ uhā€¦ calling environment. Regardless, there will be subtleties that we all need to discover through experience and experiments like this.

Youā€™re right about the protocol, though: how do flakes publish a semver-compatible version number to be found. Obvious options include github (and similar) release tags, a version metadata attribute added to the flake schema, and more.

Iā€™m not sure where flakehub is currently getting that data, and of course the point is that it can change and evolve. But the nice thing that it adds today is (what appears to be) semver match resolution in the http path to the flake, which allows more options and broader matching than current options (like following a branch that might or might not exist upstream).

1 Like