I agree with the broad strokes of this, but I want to go a bit further: I think people see something symbolic in nix’s (in)ability to abstract over platform differences.
From the inside, we know there’s no magic. Someone has to get cut by a platform difference, figure it out, and fix it.
But from the outside, I think people semi-fairly read our success or failure on this point as saying something ~profound about the power/ability of nix’s conceptual model to actually eat the world.
(I’ve said elsewhere that I see nix as having a bit of an ice-nine property. It wants to restructure much of what it interfaces with. We can either help, or admit that the interface is out of scope.)
Edit: tbf, I think our ability to handle cross-compilation is a similar signal of this ability, though I think fewer people pose that question.
I’m using nixpkgs on fleets of 4 systems: aarch64-darwin, aarch64-linux, x86_64-darwin, x86_64-linux. The darwin systems are running macOS, the linux systems are running Ubuntu. In terms of support, in my experience the darwin nixpkgs systems are about as well supported as the linux nixpkgs systems when used on a non-NixOS linux distributions. I do not use NixOS on those fleets.
All this to say that darwin support, in fact good darwin support, is something I care about a lot.
I see a lot of points in favor of darwin support and how we can make it better have already been made on the thread so I won’t reiterate them. I’m especially happy to see we’re doing some work to make darwin bootstrap / stdenv a bit more compact and self contained, I’ve definitely ended up changing the bootstrap while trying to fix something unrelated in the past, and had the suspicion the ease with which you can cause nixpkgs-wide rebuilds on darwin was one of the factors for long queue times for darwin jobs. I work in Australian time zone so actually I see the darwin queues be the same as the linux queues a lot of the time (for example at time of writing they’re all at around 8k jobs), but sometimes they jump up to 100k jobs and take days to drain.
I’ll bring up one point that I brought up on the RFC you linked from a while ago: having a system that isn’t linux and maintaining that helps nixpkgs crosssystem compatibility beyond that specific system, and helps prevent system specific idiosyncrasies from being dependended on / assumed. So having darwin as tier 2 support helps the support of any other non-linux system. In fact it’s also possible that it helps support of non-NixOS linux systems, given that darwin is the only supported non-NixOS system, but I don’t have specific examples of that.
Alas, the changes probably won’t help there. LLVM is a big dependency, and newer versions are definitely not smaller than older ones. My goal with the decoupling is to remove the need to bump bootstrap tools just to update the stdenv. As long as the tools are good enough to rebuild cctools and LLVM, that’s all it really needs. The other changes are more for maintainability.
I reworked the Darwin stdenv to follow the Linux stdenv’s patterns where it makes sense. Using assertions to catch mistakes during evaluation has been a major boon to my productivity. I don’t know that I’d have been able to make some of the changes I needed to make otherwise. I’ve also tried to be liberal with comments explaining what it’s doing. My hope is these changes will make working on the Darwin stdenv more approachable.
I don’t remember off-hand, but if you’re willing to comb through old Nixpkgs Haskell status reports, you’ll definitely find some status reports where there are huge numbers of things failing on Darwin for strange reasons. The status reports are generated every 6 hours based on the state of the haskell-updates jobset on Hydra. If you look through old commits you can find past status reports:
Just to make sure that there is no miscommunication here: I am not saying we should drop Darwin support, or that Nix shouldn’t be used on Darwin machines or something. In fact I am totally in favor of supporting more platforms, including Darwin and WSL, if that helps some people.
And wanting these things is fine, but in the end somebody has to go ahead and put the work into it, because I won’t. This is up to the people who are invested into these platforms. (And some people already do great work here, thank you for that.) Note that Darwin and WSL are a bit special here because they are proprietary ecosystems; in contrast to other platforms for which I have no problems spending my time to some extent.
Platform support tiers are not a “goal” as in saying “yes we want this”, but they are a promise, a contract for developers and users about guarantees and expectations. And currently, Nix on Darwin does not fulfill the promises of a Tier 2 platform, and subjectively it has not improved by much in the last two years, especially regarding CI infrastructure. This makes Darwin effectively a Tier 3 platform with channel blocking.
Do you have some links to that? Given that channel blockers are a major point of contention, I’d be very interested in that. Unburdening maintainers who don’t use Darwin is kind of the goal though, and ideally the Darwin team would step up to fill that gap.
I totally agree that this is a pain, and that our pinging infrastructure needs to be worked on here. Unfortunately I don’t see many good solutions apart from “moar CI”, at the very least for evaluation.
Unpinning was necessary as we had many critical bugs and the release ongoing, I just repinned it.
As a release manager, Darwin maintainers interaction was suboptimal, I didn’t feel there was anyone from the team there to help the release management team and drive it. It was extremely frustrating.
My goto example is 🦦 NixOS 23.05 — Feature Freeze & Release Blockers · Issue #224457 · NixOS/nixpkgs · GitHub
Darwin maintainers didn’t bother answering the call…
FWIW, I’m not saying it needs to be a permanently pinned issue. I realize it’s valuable real estate. I’m not sure how to reach more potential Darwin maintainers though.
That’s not a matter of not bothering. That’s about no one wanting Darwin issues blocking the NixOS release, combined with the fact that no one feels like they have the authority to speak for the entirety of Darwin maintainers. The darwin-maintainers team is like the Nixpkgs Maintainers team in that regard, not like the Haskell team for instance. It’s not a coordinated group with any structure or hierarchy.
That’s something for the Darwin maintainers to figure out .
I find this to be a weak argument honestly, sending multiple Darwin messages on this thread is probably not a big deal IMHO, it’s built for that. Silence is worse than noise on those aspects.
Honestly, I think that should change, someone has to be owner / take responsibility for it as we can see that it doesn’t work without it.
Though, I don’t think you need hierarchy or structure to go and say “hi, darwin maintainer here, we don’t see anything problematic, or we do have those issues”, release managers gets the final call here.
Otherwise, ignoring me is just making me more annoyed when I see potential release blockers and I have no idea how to evaluate them because I do not use Darwin.
I even suggested that Darwin maintainers should probably work on electing/choosing their Darwin release manager because of that.
But in all the cases, this @NixOS/darwin-maintainers doesn’t work very well in my experience, we ping them, but they don’t react in a timely fashion usually IIRC, which ends up putting the churn on annoyed Linux users who are blocked on Darwin related thing.
(and this was probably one of my biggest annoyance as a release manager for 23.05.)
And get me right: While I don’t care about Darwin, I do care about us providing Darwin for everyone and I can sympathize with people running Darwin (not necessarily by choice), though, I feel like:
The popularity earned by Darwin giving us NixOS contributors does not translate in significant manpower for Darwin maintenance or Linux contributions (please prove me wrong with data, I know I nerdsnipped someone on looking at this more seriously with data, so take this statement with a salt of grain.)
Therefore, I do not identify Darwin as an important aspect of the NixOS project beyond “users which are not contributors” metric, which is fine for people who can make money off this and invest in the project to improve the maintenance story.
Currently, the infra story (on ofborg side), the infra story (on community side, 1 remote builder provided by community members, I don’t even know if @winter pay electricity cost, etc. for everyone else) are insufficient in general. Some community members went and had some money to work on this project, I offered my own informal colo space here: macos build box for community use · Issue #493 · nix-community/infra · GitHub — again, I cannot be committed extremely with something I do not care about. If people built appliances, send to me in datacenter-style, I can host for them and if some crowdfunding funds the electricity cost, this is amazing. If not, you can also fund clouds to provide you with their spare Mac Mini (which are heavily quota’ed and in large demand), etc., etc.
But I think this post is the wakeup call for everyone who cares about Darwin to organize a plan to make this situation better before running into reopening the downgrade tier RFC.
I heard someone hyperbolically call the Darwin Maintainers team the embodiment of the bystander effect … I think your comparison to the Nixpkgs Maintainers teams is appropriate, as the Darwin Maintainers team is getting too large for simply pinging it like one would for the Haskell team. So we need an alternative here, and I think adding a bit of hierarchy to the Darwin team would probably not be a bad idea. Maybe have a label to add instead of a team ping and then there is some automation which helps triaging and assigning individual people or something?
It’s been established that if a response to a ping is not forthcoming Darwin-breaking changes can proceed. We can’t have 50+ people confirming they’re not able to help or not interested in a certain package.
Personally, I check every ping and if it’s important or trivial I act, otherwise I remain silent. If explicit negative signals are preferred I can try to be louder. It’s just that that’ll lead to kicking other Darwin maintainers’ shins when they’re already on the case.
I’m not sure a tag will improve matters since maintainers would have to actively seek it out and we don’t have that level of inertia yet AFAICT.
Same, I also check every ping, and only respond if I have some idea or context to jump in. Some of the comments make it sound like this happens all the time but looking at my email there’s only 10s of PRs per month where @NixOS/darwin-maintainers gets pinged (plus however many PRs where I ping it, since I don’t have email notifications enabled for my own messages), there’s really not that many.
I mentioned in the RFC before that darwin is the only non-linux non-gcc non-glibc platform with substantial support. How important is that capability? It might not be, and linux gcc glibc support would benefit greatly from not having the extra complexity of supporting other OS / kernels, c compilers and c stdlibs.
If it is important, then I think we can use that as a way to reduce the burden of darwin development in nixpkgs: linux + clang would cover llvm / clang, linux + musl would cover non-glibc, and freebsd or illumos could cover a separate os (incidentally they’re also part of the BSD family so they would help some aspects of darwin support). All three of those don’t require any specialised hardware and are not related to proprietary software.
If we had coverage of those three axis along which the current tier ~2 darwin systems are different from the tier ~1 linux systems, I believe the remaining darwin specific issues would be far fewer, and much more likely to be specific to darwin alone.
Note this isn’t an entirely practical suggestion. All three of those are currently less supported than darwin, and it’s possible any of three would be more effort to support than darwin, let alone all three of them together. But I want to raise it to highlight how the issue isn’t limited to darwin, but applies to all non-linux non-gcc non-glibc systems.
I think darwin-maintainers cannot work like the ecosystem specific maintainer teams, and it’s unreasonable to expect that it could: since it’s a platform and not a subset of packages that are related to each other it cuts across all of nixpkgs, there’s no common programming language / stack / common large subset of dependencies between the packages (darwin stdenv is large, but it’s still very small compared to the entire dependency closure of many of the packages).
True. At our team, we are some Linux Nix users, Linux non-Nix, and some Mac users (again, some of them using Nix, some not). Some of the Linux users maintain a cross-platform development environment for all Nix users, hoping that it will work out of the box for the Mac users. Debugging Mac problems is really annoying because Linux Nix devs do the main development in that project, while the Mac users use it for data analysis and other tasks. I would not expect our Mac users to become actively involved in nixpkgs development any time soon, but they appreciate that there is a common setup for everyone that they don’t need to maintain.
Promoting Linux+musl specifically might be feasible — it’s possible to run a complete x86_64-unknown-linux-musl NixOS system, and I and others try hard to prevent regressions.
I’m not sure how much glibc vs musl issues are likely to help Darwin, though…
I think this is a really, really good point. I’d actually question why we have that team in the first place when we don’t have a linux-maintainers team either.
It’d make much more sense to have a darwin-core team which is knowledgable about the Darwin stdenv, Frameworks and other low-level Darwin tools required for basically every Darwin package. They’re the ones who would need to give a sign-off on a release.
I guess darwin-maintainers could continue to exist as a “hey Darwin people, could any of you take a look at my PR that broke on Darwin and fix it?” team though.
Note that I would not be shocked if we had a @NixOS/linux-core which is basically a Jack of All Trades team which can go and re-ping the right persons and handle the triage for some situations where there’s absolutely no idea from where to start on.
We don’t have so many problems in that area currently, but if it was needed, we could completely have one.
I feel like what I am asking for usually is someone who knows how to triage Darwin stuff and redirect it to the right people, so a @NixOS/darwin-core would probably be awesome.
Some people would argue that BSD is not copyleft and would not want to touch it.
But anyway, to maintain BSD stuff, there would need some serious efforts to bootstrap it, and I don’t see necessarily who is going to do this in the proposal, but I would be extremely happy if more people beyond the awesome folks already working on *BSDs and non-glibc in nixpkgs.
Would Darwin “core” maintainers commit to work on it?
As a “Linux” maintainer, I feel we have enough hardships shipping a reasonable Linux system, it’s not too bad, but I want to focus on getting all of this right before moving to, working on *BSD (which I have on my long term todolist).
Like, I’d prefer to touch “non-gcc” and “non-glibc” for now, “non-linux” can also be done on Linux itself (see the work on UEFI cross-compilation and the question “how to have a cross-compiled thing for a boot environment which is != from booted environment can be used in a Nix package” is yet unresolved AFAIK.)
When this situation will be better, I feel like it can become a goal for me to consider the other capabilities.
But, this is a “me consideration”, people are free to kickstart Darwin or other things, though I don’t think it’s an inherent goal that Linux maintainers has to share beyond basic respect and non-breakage, I remember often being told “you should use != rather than == to filter broken platforms” and whenever I read expressions, I see that this “non-linux non-gcc non-glibc” philosophy has turned into a “darwin” philosphy mostly as you can see that sometimes it’s really platforms.linux ++ platforms.darwin; without any consideration for BSDs or any other system, which is sad .
@NixOS/darwin-core as a team that maintains darwin stdenv, not darwin support for non-stdenv packages. This additional focus will make the expectations clearer for other contributors.
Make linux-glibc-clang and linux-musl-gcc tier 2 supported platforms. This will ensure that there is some degree of libc and c compiler agnosticity in all derivations, regardless of darwin support. It will make working on this agnosticity aspect more accessible to linux contributors since it won’t require usage of non-linux systems. It will narrow the scope of darwin specific issues to darwin and non-linux issues alone, excluding non-glibc and non-gcc issues.
Do we think these two changes would help / be useful in their own right?
Personally I think 1 is useful, but I don’t consider myself knowledgeable enough to be part of that proposed team so I’d like to hear from the people who are.
2 is iffy: I can see linux-musl-gcc being useful on its own, but linux-glibc-clang doesn’t sound very useful, and adding new systems increases the load on our infrastructure, both compute and binary cache storage.
How is darwin support implemented anyway? Is there some standard method of doing it that doesn’t involve “if architecture is mac m1, do this, else if architecture is x86_64, do that, else if …” in one package file? Because should that exist, then the CI could be setup to trigger linux builds only for changes to linux files, darwin build for changes to darwin files, …, and trigger builds for all platforms should common files be changed.
In this manner, it would be possible to split out libs, packages, and other things into platform specific files/directories as needed and teams could make adjustments where needed without impacting other teams much.
For example, should there be a PR that modifies a common lib function and that breaks darwin support, they could split out the old function to a darwin specific file that’s automatically used by darwin builds, while the new implementation stays in the common library.
This entire discussion however begs the question: should everything stay in nixpkgs? Because it’s bound to happen (and probably has before), that changing a linux specific change breaks darwin support, the author doesn’t have a darwin machine, and there’s a dilemma: merge and break darwin or find darwin maintainer to fix then merge.
I’m not sure about the right strategy here whether a nixpkgs-darwin, nixpkgs-openbsd, etc. should be created and have separate teams handle that, or if the mono repo should continue exist. Should it continue to exist, a method should exist for platform teams to work in the same repository without treading on each other’s toes.
Of course, if I have completely missed the mark and a method of working does already exist, I’d be curious to see it. It would be an interesting read.
.
.