-
Conditional settings — and also patches that only change anything on a single platform — are indeed put into the single expression.
-
After evaluation one can see for which platforms things have changed. One can evaluate for another platform; OfBorg doesn’t use that, though.
-
I suspect that version bumps might be the majority of changes — these change things for all platforms, and there is a desire to reuse the information about updates between all platforms at least when updates work without substantial changes (which is a large fraction of all updates).
-
Having multile versions for more stuff to be able to hold back the bump for some rev-deps or for some platforms would probably help with the workflow of bumping Linux now and letting Darwin catch up without breakage. But unlike conditionals, unlimited version proliferation does meet some opposition.
Darwin has its own stdenv that provides clang, libc++, etc. It also provides Darwin-specific libraries and frameworks in the darwin attrset. When a package builds on Darwin, it’ll get the Darwin stdenv. If a framework is needed, it can be added to buildInputs conditionally.
That’s usually enough to let a package build. Sometimes there are Linux-isms that need to be undone or worked around. Darwin does have some technical debt and differences that can make things difficult at times. I wish that cross-building to Darwin worked and testing were easier for non-Darwin contributors.
Cross-building will always be more difficult than native builds. I see it as nice to have, but probably not worth maintaining for most of the packages (huge amount).
Could you point out where that’s done? Maybe some of the logic could be used to separate packages by their platform and/or arch in order to get a handle on this:
It doesn’t seem like a good separation of concerns, IMO. Having a build script or expression with a bunch of lib.optional and if/elseif/elseif/.../end might be less legible and different to follow than “look in this file for modifications specific to your platform”.
Nixpkgs is not that good with overrides, to be honest, so maybe it’s easier this way…
(Also, there is a hope some Darwin fixes are removing assumptions that we want removed on Linux, too, if the code is already written)
It’s in the definition of the stdenv. Which one is used depends on the local system. There are separate stdenv definitions for different platforms because their bootstrap processes are so different. Some packages have separate derivations for different platforms, but the changes are not usually big enough to require that. A simple conditional (e.g., lib.optional) is usually adequate for most packages.
I believe that in most vast majority of cases we do not want to separate package expressions based on linux vs. darwin.
Probably, but shouldn’t we have a clean, standard way to do so when it is needed without adding a bunch of ifs and lib.optional?
You can do it easily, e.g.
package = if stdenv.hostPlatform.isDarwin then foo else bar;
(which can be imports of separate files, to keep this part small)
I hadn’t considered this option. Maybe it would be better to just split them consistently into the darwin package set?
It currently has 125 packages, but if it’s consistent, that would also provide a better user experience. “I’m on darwin, where could I find darwin packages? Let me just type darwin in the package search”.
There could easily be a function that populates the darwin package set with packages from the global set unless they already therein. Thoughts?
It could also be just that simple. It would have to be documented too in a guide for people contributing to darwin packages 
Does it have to be in an RFC to be standardized?
Edit: it just occurred to me that this would be very specific to darwin and lead to more ifs and else for other platforms (openbsd for example) and archs. So maybe something a little more advanced, but flexible could be used in its place. Don’t know if nix lang supports structural pattern matching like python, rust, or something similar that would allow handling different cases.
For example
match platform, arch {
"darwin", * => darwin_package,
"openbsd", "x86_64" => openbsd_x86_64_package,
* => default_package
}
I don’t see how it solves the problem we are complaining about.
The problem is not that Darwin fixes completely clutter the expressions (there are cases where splitting is worth it — it can be done in those cases, it is sometimes done). I don’t think Darwin users’ main complaint is discovering package status via search.nixos.org.
The problem is about the impact of Linux-motivated changes on the Darwin side. On the Darwin side, there is obviously interest that the most interested and capable Darwin maintainer gets magically notified whenever there is damage dealt, and maybe there is a channel that always skips major Darwin breakage caused by Linux-side updates. On the Linux side, there is obviously interest to avoid needing to care about «What The Hell Were They Thinking At Apple While Breaking All The APIs Again».
Linux side kind of has its own issue like that: glibc upgrades etc. might break a lot of stuff. The workflow is based on merges in a loop staging → staging-next → master → staging. I guess a similar workflow for darwin with master → staging-darwin → master-darwin → master could work, but that would need a lot of CI throughput, which is lacking even for current, possibly smaller, needs.
Maybe I’m missing something here, but how does splitting packages not solve this? If linux maintainers work in their realm and darwin in theirs, how will linux motivated changes get onto darwin’s side and break things?
The darwin package set is for platform-specific packages. I don’t think it makes sense to put cross-platform packages there just because they require a bit of customization
In fact, I am working to remove some packages (libiconv and ICU) from that package set because the upstream versions are also available in nixpkgs, and maintaining separate derivations increases the load on Darwin maintainers unnecessarily (especially since the upstream versions also already build on Darwin).
I see. Thank you for the context 
Well… there are various packages that are mass rebuilds for both linux and darwin. So I can’t see how we would update and patch those. Like, do everything twice? (on copy&pasted package expressions)
If there are proposed Darwin patches in time for inclusion, they can of course be integrated just on staging. During staging only a small core of Darwin packages have a chance to get into shape (based on the schedule), and maybe not even that.
Fixes that need to be done for darwin-staging but cause Linux-side mass rebuild, these get done with a platform conditional. Whenever the package is touched on Linux, maintainers might choose to reintegrate some of those unconditionally.
Large Linux-side rebuild happens during staging phase (without large Darwin-side rebuild yet), large Darwin-side rebuild happens later during darwin-staging phase.
In my experience the common source of darwin specific issues that linux contributors hit is package upgrades, not large stdenv changes. It’s more a “I managed to get my derivation working with this very gcc / glibc / linux specific patch. How do I make an equivalent very clang / libSystem / darwin specific patch?”.
Since linux stdenv and darwin stdenv are quite different, they use different libc, different c++ standard library, different c compiler, it’s not as common for changes to one to affect the other, and the people making changes to one of the two are usually knowledgeable enough about the platform specific issues to fix them up. They might not be knowledgeable enough about package specific issues that happen due to large changes, which can result in the issues raised in Nixpkgs's current development workflow is not sustainable, but that’s a different matter.
If the upstream doesn’t support darwin, there’s no expectation that the package in nixpkgs support darwin. And upstream darwin support also uses clang and libSystem just like we do. So generally the issues are due to the nixpkgs specific patching required. If the patches are in the part of the code that is in common between linux and darwin upstream, then the same patch will likely work for both. If the patches are in the part of the code that is specific to one of the two platforms upstream, like specific configure flags / compiler flags, then they will have to be platform specific.
Here’s an example where a linux specific fix needs a darwin equivalent: default.nix - NixOS/nixpkgs - Sourcegraph.
Maybe a linux contributor updating a package will figure out how to use LD_LIBRARY_PATH in the build to make it work, but they might know about the darwin equivalent, DYLD_FALLBACK_LIBRARY_PATH.
Here’s an example where similar, non-system specific patches have to be applied to multiple system specific files in the upstream: https://sourcegraph.com/github.com/NixOS/nixpkgs@42b5817e6b94974614cc9d0f803c14551274d6c5/-/blob/pkgs/tools/networking/tuntox/default.nix?L46%3A1-57%3A1.
There’s also different dependency closures across the two systems: maybe the linux build requires openssl but the darwin build uses apple_sdk.frameworks.Security, or the linux build uses qt but the darwin build uses apple_sdk.frameworks.AppKit. Usually the more upstream supports macOS natively, the bigger the divergence.
Any time a linux contributors hits anything like this, they don’t have the system available to develop these darwin specific changes, no matter how small or simple they might be, and even if they wanted to have one, it would require buying specialised hardware. macOS virtualisation is only supported on macOS hosts, and I’m not sure how feasible it is to run a non-macOS darwin system. Even if they have a good idea of what the change required is based on other packages or comments by reviewers, it might take multiple rounds of waiting for ofborg to build the derivation to confirm the correct syntax / specifics, and the queues and build performance might not be good.
I said earlier that in my experience ofborg performance for darwin is not too different from linux, but at the same time I have systems for both darwin and linux development, so I don’t have to wait on ofborg for either. Someone who only had a darwin system would hit the same issues and frustrations as someone only having a linux system, but at least virtualising linux is easy enough and well supported in most cases.
(merged into previous post)
(merged into previous post)