Thanks for the offer, @pjjw, but the Python issue was tracked down to intentionally using python3Minimal for cacert to workaround an infinite recursion issue with mailcap. This has since been rolled back and I believe it was fixed by using fetchurl instead of fetchzip for mailcap, but this hasnāt been confirmed.
Iāll let you know if the need arises again : )
Continuing to bump source releases. hfs is the only one so far that I havenāt been able to update, newer versions miss headers present in the SDK. I managed to get more of shell_cmds building. Only sudo is still missing but in later versions it may require entitlements so we probably wouldnāt be able to keep it building anyway. I also made some changes to our header checks to include checking for symlinked headers, which are common in the SDK.
The Libc bump is more involved because it used to be an amalgam of three versions and Iām dropping all the old ones, which requires digging through the sources for missing headers. So far it seems viable to drop the older versions.
Iām down to only a handful of source releases that still need bumping. Libc still requires one older version because Apple stopped shipping certain headers. The pkill and pgrep commands of adv_cmds still do not build, but colldef and mklocale do. One of the bigger challenges was network_cmds, it depends on OpenSSL. However, Appleās OpenSSL for macOS 10.13 was at version 0.9.8. The network_cmds in Nixpkgs is currently built with OpenSSL 1.0.2 but the new version wasnāt compatible anymore. Rather than go back to Appleās older version, I patched network_cmds for compatibility with OpenSSL 1.1.0. Eyes on this would be appreciated.
In other news, @reckenrode 
made it possible to use the 11.0 SDK on x86_64-darwin! This was prompted by their work on MoltenVK and additionally motivated by Go 1.18 requiring a newer SDK, not to mention various other projects like Python and Qt. For now the intent is for this to be used sparingly, only for cases where itās unavoidable.
Iāve bumped all of the Apple source releases, configd being the main missing piece. For now we will stick with SystemConfiguration from the Apple SDK as a substitute. This time around, system_cmds was the most troublesome project. I ended up having to drop gcore, lskq, lsmp and zprint but managed to fix the builds for dirhelper, dmesg, dynamic_pager, ltop, pagesize, proc_uuid_policy, taskpolicyandvm_purgeable_stat`, it remains to be seen whether those are requirements for any packages.
The next step is to do a full rebuild on Hydra to check how many failures are introduced by these changes. And then things need to go through review. Iād especially appreciate more eyes on the changes to network_cmds related to OpenSSL. @veprbl already pointed out an initialization issue but Iām sure this can use more scrutiny.
One current technical hurdle is Hydra is not currently building bootstrap-tools, but this is being worked on in PR #183072.
Latest evaluation still has many failing jobs. I included one too many changes from darwin-stubs and those will only start working once the corresponding Nixpkgs PR is merged. The deadline for changes to release critical packages for 22.11 is coming up in less than two weeks. This doesnāt leave enough time for thorough reviewing. So a bit of a false start but on the bright side this should be the last release window missed.
Thanks to @a-m-joseph for fixing the bootstrap-tools SNAFU in PR #183072, this change is currently making its way through staging. The bootstrap-tools are failing on aarch64-darwin, which will block nixpkgs-unstable so there is more work to be done.
The availability attribute I started using here is not compatible with GCC. This didnāt really come up before because Darwin stdenv uses Clang. However, many packages in Nixpkgs do use GFortran, which is really GCC so it did come up in the Hydra evaluation. The way to support availability macros for GCC is simply to go with Appleās older approach of having macros for all the various combinations of versions.
I want to highlight @stephankās work bringing Swift to Darwin. Itās a big change so more eyes to help along the review would be welcome.
I finished implementing the old-style macros (comparison to trunk) and GCC and GFortran (except for 6 and older) now build. There are still many failing builds, some of these seem related to the builds running under Rosetta2 but there are other problems. RustC is failing to build (a bootstrap stage compiler is exiting unsuccessfully), Pythonās requests is failing to build due to CFFI (Could this be related to #187636, since these builds are running under Rosetta2 on aarch64-darwin?), Libepoxy is another common dependency failing due to a segmentation violation. So lots of issues to figure out, if anyone feels like they may be able to help, feel free to reach out.
This month marks an end to my Nix macOS sponsorship. Iām grateful for having had the opportunity to focus on Nixpkgs development for macOS. Going forward I wonāt be able to spend as much time on this but I do want to finish the bump. If youāre short on time but want to help out the situation for Nix on macOS, consider contributing financially to the Open Collective.
Happy Holidays and a great 2023 to everyone!
Thank you for all the work youāve done thus far!
Absolutely great, thanks so much for what you have done and the reports!
Looks like you forgot to finish your sentence 
I added a my contribution on open collective. Itās not much but a little bit per month will surely help
Holochain is graciously sponsoring improvements to the Darwin situation in Nixpkgs for six months. They leverage Nix for development environments and value being able to do so across both Linux and macOS.
@reckenrode has been doing great work on the stdenv with the goal of decoupling it from the bootstrap-tools. So far the LLVM version in the Darwin stdenv was tied to the version included in the bootstrap-tools. The rework of the stdenv makes it possible to bump the LLVM in the stdenv independently from the bootstrap-tools. This took a lot of work so I focused on unblocking it. The rework PR got merged in the meantime and more workās underway to bump LLVM.
There was another discussion on how some maintainers experience Nixpkgsā Darwin support as a burden. My main takeaway is that people want a team to contact for Darwin issues that block Linux channel progress.
Iāve also resumed work on the stdenv bump. The source releases are a tangled mess of things (sometimes mutually) depending on each other. So we donāt want to bump individual source releases incrementally. But the full bump grew to over 130 commits (this is after some householding) and thatās a bit rough for the review process. So Iāve settled on a plan of introducing new versions of the source releases alongside the current ones. This way I can split the bump into small PRs focussing on individual releases and then when all of the releases have been merged, we can switch the entire set over all at once.
The stdenv LLVM bump is progressing. Shout-outs to @reckenrode and everyone chipping in!
Iāve continued splitting the SDK bump into smaller branches. One snag has been bumping the SDK itself, which dtrace depends on (in later versions we might be able to drop that dependency again), leading to a bit of a circular dependency situation because the SDK requires the stdenv and part of the stdenv requires the SDK.
The stdenv LLVM bump is still in need of more reviews. @reckenrode is also working on fixing cross-compilation from aarch64-darwin to x86_64-darwin.
All the dependencies of the bootstrap-tools are split into branches. The other source releases should be buildable against the new bootstrap-tools and avoid a needless rebuild later. I have a patch for issue #150655, however, it makes the binutils build fail and thatās pretty low down in the dependency chain and Iāve been focusing on the prerequisite PRs for the LLVM bump instead. Iām also trying to add the Compression framework to the SDKs, which is needed for newer system_cmds, but running into needing newer darwin-stubs. Either those need a new release or we can switch to fetching the repo instead.
A bit of extra context for the system_cmds update: what Iām trying to do is see just how new we can go since the _cmds packages arenāt actually shipped with a particular SDK release. Iāve got most of system_cmds 970.0.4 building, which is the release corresponding to macOS 14 (but using the default SDKs in nixpkgs). After that, I want to look at the other releases.
Iām also replacing the build systems with Meson. Some of the packages try to use Appleās Xcode projects while others have a custom build phase. Apple keeps changing the structure of things, and xcbuild is no longer maintained anyway, so I gave up on trying to make it work.
The stdenv LLVM bump was merged. It introduced some issues but theyāre being addressed. The cross-compilation PR is still open, but, currently fixes required due to the bump are prioritized.
The patch for issue #150655 broke the binutils build because of unused argument warnings, which cause its configure script to misdetect the presence of standard library headers. Iām still looking into the linker-side of things, the current patch only addresses the compiler-side. I will also continue with the Compression framework, to unblock newer system_cmds and then the stdenv work.
I would also like to mention two things: #265103 and overrideSDK.
The first PR will make CoreFoundation the default over the open-source CF on x86_64-darwin. It is planned for the next staging cycle. The change is being made due to compatibility issues on macOS 14 (possibly due to CorrFoundation hardening).
overrideSDK is a replacement for the apple_sdk_11_0.callPackage pattern. It takes a stdenv and an SDK version string, and it provides an adapted stdenv that uses frameworks from the requested SDK including propagated ones. It also ensures that a compatible xcbuild is used. overrideSDK can also be used to change the deployment target (in addition to the SDK version or just that).
overrideSDK is currently in staging-next and gaining features as needed. Once additional SDKs are introduced and the SDK hierarchy is standardized, it will be updated to support using SDK-specific libraries (such as simd).
The CoreFoundation PR was merged today as part of the merge of stage into staging-next (and the restart of builds). Most of the existing CF-centric stuff (hooks, etc) was left in place (in case the PR needs to be reverted but also to keep the change small), but I expect it will be cleaned up and removed for 24.05.
Progress on the cc-wrapper fixes is slow, the large rebuilds make it hard to iterate. The Compression framework bump is currently stuck on a problem with the OpenDirectory framework.
Iāve opened an overview issue for the first source release bumps. The intent is to introduce the new source releases alongside the old ones to enable review in small units and then switch over when everythingās merged. Iāve left the PRs in draft for now in case we can come up with a better way than the nested attribute set in the source releases expression.
Have you considered using ccache?