As far as I can tell, the command-line tools (not sure about Xcode) make an exception only for the open-source components (and I have no idea how that works when the open-source components are incomplete or unavailable). What about the other stuff though? How does that work for e.g., Metal or CoreImage?
April 2022
Something that stood out with the _scproxy issue was the involvement of a bootstrap stage Python in the bootstrapped-pip build. This raised my suspicion that something was wrong with the Python build. Turns out there were errors preventing _scproxy.c from being compiled but these errors were silently swallowed and did not result in a failing build.
The reason for the errors was an issue with the Availability macros. Apple stopped updating them in the open source releases, probably because of the combinatorial explosion of macros with each new version of macOS. Theyāve since switched to newer macros, which are backed by the availability attribute in modern Clang and GCC.
My fix is redefining the old macros using the new-style availability attribute. This finally fixed the Python build and completed the substitution of configd by SystemConfiguration from the SDK. Now I can continue bumping Appleās open source releases.
16 Likes
May 2022
With configd out of the way I was finally able to focus on bumping the actual Apple open source releases we rely on to construct our own source-based SDK. This went swimmingly. Iāve updated about half of the source releases and tracked down some of the headers we were missing, so our SDK will be more complete. So far I havenāt encountered any major, new roadblocksāknock on wood.
19 Likes
@toonn: I see these errors popping up again, after switching default python from 3.9 to 3.10 (currently on the staging-next branch). Very many packages seem to be transitively broken by that. Perhaps you could have a look?
1 Like
Has anyone dared to try upgrading to the Ventura developer beta yet? Weāve gotten lucky with the last 1 or 2 upgrades and things have mostly worked out of the box, but prior to that itās often been a very painful process.
1 Like
Weāve been talking about this a bit in #macos:nixos.org. Iām looking into this today but thereās only so much I can do because I donāt have access to aarch64-darwin hardware.
What I can already say is that I donāt think the errors are related to the ones I observed. As aarch64-darwin is using SDK 11.0.1 it shouldnāt have issues with the Availability macros.
2 Likes
Right now, x86_64-darwin seems to be broken with the same error message.
1 Like
would you like access? happy to provide
Thanks for the offer, @pjjw, but the Python issue was tracked down to intentionally using python3Minimal for cacert to workaround an infinite recursion issue with mailcap. This has since been rolled back and I believe it was fixed by using fetchurl instead of fetchzip for mailcap, but this hasnāt been confirmed.
Iāll let you know if the need arises again : )
1 Like
June 2022
Continuing to bump source releases. hfs is the only one so far that I havenāt been able to update, newer versions miss headers present in the SDK. I managed to get more of shell_cmds building. Only sudo is still missing but in later versions it may require entitlements so we probably wouldnāt be able to keep it building anyway. I also made some changes to our header checks to include checking for symlinked headers, which are common in the SDK.
The Libc bump is more involved because it used to be an amalgam of three versions and Iām dropping all the old ones, which requires digging through the sources for missing headers. So far it seems viable to drop the older versions.
13 Likes
July 2022
Iām down to only a handful of source releases that still need bumping. Libc still requires one older version because Apple stopped shipping certain headers. The pkill and pgrep commands of adv_cmds still do not build, but colldef and mklocale do. One of the bigger challenges was network_cmds, it depends on OpenSSL. However, Appleās OpenSSL for macOS 10.13 was at version 0.9.8. The network_cmds in Nixpkgs is currently built with OpenSSL 1.0.2 but the new version wasnāt compatible anymore. Rather than go back to Appleās older version, I patched network_cmds for compatibility with OpenSSL 1.1.0. Eyes on this would be appreciated.
In other news, @reckenrode 
made it possible to use the 11.0 SDK on x86_64-darwin! This was prompted by their work on MoltenVK and additionally motivated by Go 1.18 requiring a newer SDK, not to mention various other projects like Python and Qt. For now the intent is for this to be used sparingly, only for cases where itās unavoidable.
19 Likes
August 2022
Iāve bumped all of the Apple source releases, configd being the main missing piece. For now we will stick with SystemConfiguration from the Apple SDK as a substitute. This time around, system_cmds was the most troublesome project. I ended up having to drop gcore, lskq, lsmp and zprint but managed to fix the builds for dirhelper, dmesg, dynamic_pager, ltop, pagesize, proc_uuid_policy, taskpolicyandvm_purgeable_stat`, it remains to be seen whether those are requirements for any packages.
The next step is to do a full rebuild on Hydra to check how many failures are introduced by these changes. And then things need to go through review. Iād especially appreciate more eyes on the changes to network_cmds related to OpenSSL. @veprbl already pointed out an initialization issue but Iām sure this can use more scrutiny.
One current technical hurdle is Hydra is not currently building bootstrap-tools, but this is being worked on in PR #183072.
10 Likes
September 2022
Latest evaluation still has many failing jobs. I included one too many changes from darwin-stubs and those will only start working once the corresponding Nixpkgs PR is merged. The deadline for changes to release critical packages for 22.11 is coming up in less than two weeks. This doesnāt leave enough time for thorough reviewing. So a bit of a false start but on the bright side this should be the last release window missed.
Thanks to @a-m-joseph for fixing the bootstrap-tools SNAFU in PR #183072, this change is currently making its way through staging. The bootstrap-tools are failing on aarch64-darwin, which will block nixpkgs-unstable so there is more work to be done.
13 Likes
October 2022
The availability attribute I started using here is not compatible with GCC. This didnāt really come up before because Darwin stdenv uses Clang. However, many packages in Nixpkgs do use GFortran, which is really GCC so it did come up in the Hydra evaluation. The way to support availability macros for GCC is simply to go with Appleās older approach of having macros for all the various combinations of versions.
I want to highlight @stephankās work bringing Swift to Darwin. Itās a big change so more eyes to help along the review would be welcome.
9 Likes
November 2022
I finished implementing the old-style macros (comparison to trunk) and GCC and GFortran (except for 6 and older) now build. There are still many failing builds, some of these seem related to the builds running under Rosetta2 but there are other problems. RustC is failing to build (a bootstrap stage compiler is exiting unsuccessfully), Pythonās requests is failing to build due to CFFI (Could this be related to #187636, since these builds are running under Rosetta2 on aarch64-darwin?), Libepoxy is another common dependency failing due to a segmentation violation. So lots of issues to figure out, if anyone feels like they may be able to help, feel free to reach out.
This month marks an end to my Nix macOS sponsorship. Iām grateful for having had the opportunity to focus on Nixpkgs development for macOS. Going forward I wonāt be able to spend as much time on this but I do want to finish the bump. If youāre short on time but want to help out the situation for Nix on macOS, consider contributing financially to the Open Collective.
Happy Holidays and a great 2023 to everyone!
20 Likes
Thank you for all the work youāve done thus far!
3 Likes
Absolutely great, thanks so much for what you have done and the reports!
1 Like
Looks like you forgot to finish your sentence 
1 Like
I added a my contribution on open collective. Itās not much but a little bit per month will surely help
6 Likes
June 2023
Holochain is graciously sponsoring improvements to the Darwin situation in Nixpkgs for six months. They leverage Nix for development environments and value being able to do so across both Linux and macOS.
@reckenrode has been doing great work on the stdenv with the goal of decoupling it from the bootstrap-tools. So far the LLVM version in the Darwin stdenv was tied to the version included in the bootstrap-tools. The rework of the stdenv makes it possible to bump the LLVM in the stdenv independently from the bootstrap-tools. This took a lot of work so I focused on unblocking it. The rework PR got merged in the meantime and more workās underway to bump LLVM.
There was another discussion on how some maintainers experience Nixpkgsā Darwin support as a burden. My main takeaway is that people want a team to contact for Darwin issues that block Linux channel progress.
Iāve also resumed work on the stdenv bump. The source releases are a tangled mess of things (sometimes mutually) depending on each other. So we donāt want to bump individual source releases incrementally. But the full bump grew to over 130 commits (this is after some householding) and thatās a bit rough for the review process. So Iāve settled on a plan of introducing new versions of the source releases alongside the current ones. This way I can split the bump into small PRs focussing on individual releases and then when all of the releases have been merged, we can switch the entire set over all at once.
18 Likes