Maybe I’m missing something here, but how does splitting packages not solve this? If linux maintainers work in their realm and darwin in theirs, how will linux motivated changes get onto darwin’s side and break things?
The darwin package set is for platform-specific packages. I don’t think it makes sense to put cross-platform packages there just because they require a bit of customization
In fact, I am working to remove some packages (libiconv and ICU) from that package set because the upstream versions are also available in nixpkgs, and maintaining separate derivations increases the load on Darwin maintainers unnecessarily (especially since the upstream versions also already build on Darwin).
8 Likes
I see. Thank you for the context 
Well… there are various packages that are mass rebuilds for both linux and darwin. So I can’t see how we would update and patch those. Like, do everything twice? (on copy&pasted package expressions)
2 Likes
If there are proposed Darwin patches in time for inclusion, they can of course be integrated just on staging. During staging only a small core of Darwin packages have a chance to get into shape (based on the schedule), and maybe not even that.
Fixes that need to be done for darwin-staging but cause Linux-side mass rebuild, these get done with a platform conditional. Whenever the package is touched on Linux, maintainers might choose to reintegrate some of those unconditionally.
Large Linux-side rebuild happens during staging phase (without large Darwin-side rebuild yet), large Darwin-side rebuild happens later during darwin-staging phase.
In my experience the common source of darwin specific issues that linux contributors hit is package upgrades, not large stdenv changes. It’s more a “I managed to get my derivation working with this very gcc / glibc / linux specific patch. How do I make an equivalent very clang / libSystem / darwin specific patch?”.
Since linux stdenv and darwin stdenv are quite different, they use different libc, different c++ standard library, different c compiler, it’s not as common for changes to one to affect the other, and the people making changes to one of the two are usually knowledgeable enough about the platform specific issues to fix them up. They might not be knowledgeable enough about package specific issues that happen due to large changes, which can result in the issues raised in Nixpkgs's current development workflow is not sustainable, but that’s a different matter.
If the upstream doesn’t support darwin, there’s no expectation that the package in nixpkgs support darwin. And upstream darwin support also uses clang and libSystem just like we do. So generally the issues are due to the nixpkgs specific patching required. If the patches are in the part of the code that is in common between linux and darwin upstream, then the same patch will likely work for both. If the patches are in the part of the code that is specific to one of the two platforms upstream, like specific configure flags / compiler flags, then they will have to be platform specific.
Here’s an example where a linux specific fix needs a darwin equivalent: default.nix - NixOS/nixpkgs - Sourcegraph.
Maybe a linux contributor updating a package will figure out how to use LD_LIBRARY_PATH in the build to make it work, but they might know about the darwin equivalent, DYLD_FALLBACK_LIBRARY_PATH.
Here’s an example where similar, non-system specific patches have to be applied to multiple system specific files in the upstream: https://sourcegraph.com/github.com/NixOS/nixpkgs@42b5817e6b94974614cc9d0f803c14551274d6c5/-/blob/pkgs/tools/networking/tuntox/default.nix?L46%3A1-57%3A1.
There’s also different dependency closures across the two systems: maybe the linux build requires openssl but the darwin build uses apple_sdk.frameworks.Security, or the linux build uses qt but the darwin build uses apple_sdk.frameworks.AppKit. Usually the more upstream supports macOS natively, the bigger the divergence.
Any time a linux contributors hits anything like this, they don’t have the system available to develop these darwin specific changes, no matter how small or simple they might be, and even if they wanted to have one, it would require buying specialised hardware. macOS virtualisation is only supported on macOS hosts, and I’m not sure how feasible it is to run a non-macOS darwin system. Even if they have a good idea of what the change required is based on other packages or comments by reviewers, it might take multiple rounds of waiting for ofborg to build the derivation to confirm the correct syntax / specifics, and the queues and build performance might not be good.
I said earlier that in my experience ofborg performance for darwin is not too different from linux, but at the same time I have systems for both darwin and linux development, so I don’t have to wait on ofborg for either. Someone who only had a darwin system would hit the same issues and frustrations as someone only having a linux system, but at least virtualising linux is easy enough and well supported in most cases.
6 Likes
(merged into previous post)
(merged into previous post)
Part of the conversation here could be side-stepped by introducing per-platform meta.maintainers. Let’s ignore the implementation details for now and assume that each package has a set of maintainers, that map to various platforms.
We can then use that information to declare if a package is supported for a given platform. If it’s unsupported, it doesn’t become a release blocker. If it annoys you, then become a maintainer of the package!
If we forget about macOS for a minute and think of Risc5 or other interesting architectures, the same problem exists there; not everybody has access to those machines. And the x86_64-linux maintainer might not want to wait on the Risc5 guy to come around. So I believe some mechanism like that is needed anyways.
Another benefit is that it gives users a better picture of which package is supported for which platform. And also encourages their participation if they see a package they like that isn’t supported on their platform.
9 Likes
A problem prefiguring this, though, is that being a package maintainer is neither a responsibility nor a privilege as things stand currently. (I believe there is already an endless discourse thread on this, so hopefully I’ve not re-lit that particular dumpster fire…)
4 Likes
I’d like to re-suggest splitting up Linux and Darwin into separate channels, at least for unstable.
1 Like
If you can point to specific instances of Darwin blocking Linux advancements in a significant way in recent times, I don’t see why not.
For what it’s worth, the Darwin stdenv rework PR is up. It includes changes to the open-source CoreFoundation to no longer link libcurl. It’s not possible to drop curl from the Darwin stdenv bootstrap completely (because too many things use fetchFromGitHub, unfortunately), but it simplifies the build and should hopefully make changes to curl less painful (because it never makes it to the final stdenv and won’t trip the allowedRequisites check). It should be possible to revert the SystemConfiguration patch once the PR is merged. (I mention this because several curl issues were cited in the Darwin demotion RFC.)
9 Likes
It makes contributing to Nix harder for new comers that don’t own a Darwin machine, for example my first contribution was a Graphical Program that should be supported by Mac and Linux but failed to build on Mac and not having any way of debugging this is really frustrating.
1 Like
All you can be expected to do as a Linux package maintainer is this:
platforms = linux ++ darwin;
broken = stdenv.isDarwin;
Leave it to a Darwin person to do the Darwin stuff. Reach out to the darwin people (i.e. ping @darwin-maintainers), wait a few days and if noone answers your call, broken = stdenv.isDarwin; it is.
7 Likes
Would it be possible to make a virtual darwin remote builder using GitHub - ngi-nix/OSX-KVM ?
EDIT: more stuff: sickcodes/Docker-OSX and kholia/OSX-KVM
1 Like
You could but IMHO, you can’t expect Linux maintainers to do that.
2 Likes
By “do that” you mean write such a remote builder it or use it if it already exists? See the equivalent for darwin maintainers: Nixpkgs 24.05 manual | Nix & NixOS.
Note that even if it existed it would be against the EULA for sure, and the legality of it is a grey area: https://dortania.github.io/OpenCore-Install-Guide/why-oc.html#legality-of-hackintoshing, so I don’t think we should expect people to do that.
But I still want to understand if there was a legal alternative if that could be expected to be used. For example if we had a non-macOS darwin image.
1 Like
Using it, debugging it and, by extent, being in any way knowledgable about Darwin specific stuff such as Frameworks or common issues.
You also can’t expect a Darwin person to fix Linux issues unless they’re also a Linux user.
4 Likes