Right now, x86_64-darwin seems to be broken with the same error message.
would you like access? happy to provide
Thanks for the offer, @pjjw, but the Python issue was tracked down to intentionally using
cacert to workaround an infinite recursion issue with
mailcap. This has since been rolled back and I believe it was fixed by using
fetchurl instead of
mailcap, but this hasn’t been confirmed.
I’ll let you know if the need arises again : )
Continuing to bump source releases.
hfs is the only one so far that I haven’t been able to update, newer versions miss headers present in the SDK. I managed to get more of
shell_cmds building. Only
sudo is still missing but in later versions it may require entitlements so we probably wouldn’t be able to keep it building anyway. I also made some changes to our header checks to include checking for symlinked headers, which are common in the SDK.
Libc bump is more involved because it used to be an amalgam of three versions and I’m dropping all the old ones, which requires digging through the sources for missing headers. So far it seems viable to drop the older versions.
I’m down to only a handful of source releases that still need bumping.
Libc still requires one older version because Apple stopped shipping certain headers. The
pgrep commands of
adv_cmds still do not build, but
mklocale do. One of the bigger challenges was
network_cmds, it depends on OpenSSL. However, Apple’s OpenSSL for macOS 10.13 was at version 0.9.8. The
network_cmds in Nixpkgs is currently built with OpenSSL 1.0.2 but the new version wasn’t compatible anymore. Rather than go back to Apple’s older version, I patched
network_cmds for compatibility with OpenSSL 1.1.0. Eyes on this would be appreciated.
In other news, @reckenrode made it possible to use the 11.0 SDK on x86_64-darwin! This was prompted by their work on MoltenVK and additionally motivated by Go 1.18 requiring a newer SDK, not to mention various other projects like Python and Qt. For now the intent is for this to be used sparingly, only for cases where it’s unavoidable.
I’ve bumped all of the Apple source releases,
configd being the main missing piece. For now we will stick with
SystemConfiguration from the Apple SDK as a substitute. This time around,
system_cmds was the most troublesome project. I ended up having to drop
zprint but managed to fix the builds for dirhelper
andvm_purgeable_stat`, it remains to be seen whether those are requirements for any packages.
The next step is to do a full rebuild on Hydra to check how many failures are introduced by these changes. And then things need to go through review. I’d especially appreciate more eyes on the changes to
network_cmds related to OpenSSL. @veprbl already pointed out an initialization issue but I’m sure this can use more scrutiny.
One current technical hurdle is Hydra is not currently building bootstrap-tools, but this is being worked on in PR #183072.
Latest evaluation still has many failing jobs. I included one too many changes from
darwin-stubs and those will only start working once the corresponding Nixpkgs PR is merged. The deadline for changes to release critical packages for 22.11 is coming up in less than two weeks. This doesn’t leave enough time for thorough reviewing. So a bit of a false start but on the bright side this should be the last release window missed.
Thanks to @a-m-joseph for fixing the bootstrap-tools SNAFU in PR #183072, this change is currently making its way through staging. The
bootstrap-tools are failing on aarch64-darwin, which will block
nixpkgs-unstable so there is more work to be done.
The availability attribute I started using here is not compatible with GCC. This didn’t really come up before because Darwin stdenv uses Clang. However, many packages in Nixpkgs do use GFortran, which is really GCC so it did come up in the Hydra evaluation. The way to support availability macros for GCC is simply to go with Apple’s older approach of having macros for all the various combinations of versions.
I finished implementing the old-style macros (comparison to trunk) and GCC and GFortran (except for 6 and older) now build. There are still many failing builds, some of these seem related to the builds running under Rosetta2 but there are other problems. RustC is failing to build (a bootstrap stage compiler is exiting unsuccessfully), Python’s requests is failing to build due to CFFI (Could this be related to #187636, since these builds are running under Rosetta2 on aarch64-darwin?), Libepoxy is another common dependency failing due to a segmentation violation. So lots of issues to figure out, if anyone feels like they may be able to help, feel free to reach out.
This month marks an end to my Nix macOS sponsorship. I’m grateful for having had the opportunity to focus on Nixpkgs development for macOS. Going forward I won’t be able to spend as much time on this but I do want to finish the bump. If you’re short on time but want to help out the situation for Nix on macOS, consider contributing financially to the Open Collective.
Happy Holidays and a great 2023 to everyone!
Thank you for all the work you’ve done thus far!
Absolutely great, thanks so much for what you have done and the reports!
Looks like you forgot to finish your sentence
I added a my contribution on open collective. It’s not much but a little bit per month will surely help
Holochain is graciously sponsoring improvements to the Darwin situation in Nixpkgs for six months. They leverage Nix for development environments and value being able to do so across both Linux and macOS.
@reckenrode has been doing great work on the stdenv with the goal of decoupling it from the bootstrap-tools. So far the LLVM version in the Darwin stdenv was tied to the version included in the bootstrap-tools. The rework of the stdenv makes it possible to bump the LLVM in the stdenv independently from the bootstrap-tools. This took a lot of work so I focused on unblocking it. The rework PR got merged in the meantime and more work’s underway to bump LLVM.
There was another discussion on how some maintainers experience Nixpkgs’ Darwin support as a burden. My main takeaway is that people want a team to contact for Darwin issues that block Linux channel progress.
I’ve also resumed work on the stdenv bump. The source releases are a tangled mess of things (sometimes mutually) depending on each other. So we don’t want to bump individual source releases incrementally. But the full bump grew to over 130 commits (this is after some householding) and that’s a bit rough for the review process. So I’ve settled on a plan of introducing new versions of the source releases alongside the current ones. This way I can split the bump into small PRs focussing on individual releases and then when all of the releases have been merged, we can switch the entire set over all at once.
I’ve continued splitting the SDK bump into smaller branches. One snag has been bumping the SDK itself, which dtrace depends on (in later versions we might be able to drop that dependency again), leading to a bit of a circular dependency situation because the SDK requires the stdenv and part of the stdenv requires the SDK.
The stdenv LLVM bump is still in need of more reviews. @reckenrode is also working on fixing cross-compilation from aarch64-darwin to x86_64-darwin.
All the dependencies of the bootstrap-tools are split into branches. The other source releases should be buildable against the new bootstrap-tools and avoid a needless rebuild later. I have a patch for issue #150655, however, it makes the binutils build fail and that’s pretty low down in the dependency chain and I’ve been focusing on the prerequisite PRs for the LLVM bump instead. I’m also trying to add the Compression framework to the SDKs, which is needed for newer
system_cmds, but running into needing newer darwin-stubs. Either those need a new release or we can switch to fetching the repo instead.
A bit of extra context for the system_cmds update: what I’m trying to do is see just how new we can go since the _cmds packages aren’t actually shipped with a particular SDK release. I’ve got most of system_cmds 970.0.4 building, which is the release corresponding to macOS 14 (but using the default SDKs in nixpkgs). After that, I want to look at the other releases.
I’m also replacing the build systems with Meson. Some of the packages try to use Apple’s Xcode projects while others have a custom build phase. Apple keeps changing the structure of things, and xcbuild is no longer maintained anyway, so I gave up on trying to make it work.
The patch for issue #150655 broke the binutils build because of unused argument warnings, which cause its configure script to misdetect the presence of standard library headers. I’m still looking into the linker-side of things, the current patch only addresses the compiler-side. I will also continue with the Compression framework, to unblock newer
system_cmds and then the stdenv work.
I would also like to mention two things: #265103 and
The first PR will make CoreFoundation the default over the open-source CF on x86_64-darwin. It is planned for the next staging cycle. The change is being made due to compatibility issues on macOS 14 (possibly due to CorrFoundation hardening).
overrideSDK is a replacement for the
apple_sdk_11_0.callPackage pattern. It takes a stdenv and an SDK version string, and it provides an adapted stdenv that uses frameworks from the requested SDK including propagated ones. It also ensures that a compatible xcbuild is used.
overrideSDK can also be used to change the deployment target (in addition to the SDK version or just that).
overrideSDK is currently in staging-next and gaining features as needed. Once additional SDKs are introduced and the SDK hierarchy is standardized, it will be updated to support using SDK-specific libraries (such as simd).
The CoreFoundation PR was merged today as part of the merge of stage into staging-next (and the restart of builds). Most of the existing CF-centric stuff (hooks, etc) was left in place (in case the PR needs to be reverted but also to keep the change small), but I expect it will be cleaned up and removed for 24.05.