Nix šŸ–¤ macOS Monthly

June 2022

Continuing to bump source releases. hfs is the only one so far that I havenā€™t been able to update, newer versions miss headers present in the SDK. I managed to get more of shell_cmds building. Only sudo is still missing but in later versions it may require entitlements so we probably wouldnā€™t be able to keep it building anyway. I also made some changes to our header checks to include checking for symlinked headers, which are common in the SDK.

The Libc bump is more involved because it used to be an amalgam of three versions and Iā€™m dropping all the old ones, which requires digging through the sources for missing headers. So far it seems viable to drop the older versions.

13 Likes

July 2022

Iā€™m down to only a handful of source releases that still need bumping. Libc still requires one older version because Apple stopped shipping certain headers. The pkill and pgrep commands of adv_cmds still do not build, but colldef and mklocale do. One of the bigger challenges was network_cmds, it depends on OpenSSL. However, Appleā€™s OpenSSL for macOS 10.13 was at version 0.9.8. The network_cmds in Nixpkgs is currently built with OpenSSL 1.0.2 but the new version wasnā€™t compatible anymore. Rather than go back to Appleā€™s older version, I patched network_cmds for compatibility with OpenSSL 1.1.0. Eyes on this would be appreciated.

In other news, @reckenrode :heart: made it possible to use the 11.0 SDK on x86_64-darwin! This was prompted by their work on MoltenVK and additionally motivated by Go 1.18 requiring a newer SDK, not to mention various other projects like Python and Qt. For now the intent is for this to be used sparingly, only for cases where itā€™s unavoidable.

19 Likes

August 2022

Iā€™ve bumped all of the Apple source releases, configd being the main missing piece. For now we will stick with SystemConfiguration from the Apple SDK as a substitute. This time around, system_cmds was the most troublesome project. I ended up having to drop gcore, lskq, lsmp and zprint but managed to fix the builds for dirhelper, dmesg, dynamic_pager, ltop, pagesize, proc_uuid_policy, taskpolicyandvm_purgeable_stat`, it remains to be seen whether those are requirements for any packages.

The next step is to do a full rebuild on Hydra to check how many failures are introduced by these changes. And then things need to go through review. Iā€™d especially appreciate more eyes on the changes to network_cmds related to OpenSSL. @veprbl already pointed out an initialization issue but Iā€™m sure this can use more scrutiny.

One current technical hurdle is Hydra is not currently building bootstrap-tools, but this is being worked on in PR #183072.

10 Likes

September 2022

Latest evaluation still has many failing jobs. I included one too many changes from darwin-stubs and those will only start working once the corresponding Nixpkgs PR is merged. The deadline for changes to release critical packages for 22.11 is coming up in less than two weeks. This doesnā€™t leave enough time for thorough reviewing. So a bit of a false start but on the bright side this should be the last release window missed.

Thanks to @a-m-joseph :heart: for fixing the bootstrap-tools SNAFU in PR #183072, this change is currently making its way through staging. The bootstrap-tools are failing on aarch64-darwin, which will block nixpkgs-unstable so there is more work to be done.

13 Likes

October 2022

The availability attribute I started using here is not compatible with GCC. This didnā€™t really come up before because Darwin stdenv uses Clang. However, many packages in Nixpkgs do use GFortran, which is really GCC so it did come up in the Hydra evaluation. The way to support availability macros for GCC is simply to go with Appleā€™s older approach of having macros for all the various combinations of versions.

I want to highlight @stephankā€™s work bringing Swift to Darwin. Itā€™s a big change so more eyes to help along the review would be welcome.

9 Likes

November 2022

I finished implementing the old-style macros (comparison to trunk) and GCC and GFortran (except for 6 and older) now build. There are still many failing builds, some of these seem related to the builds running under Rosetta2 but there are other problems. RustC is failing to build (a bootstrap stage compiler is exiting unsuccessfully), Pythonā€™s requests is failing to build due to CFFI (Could this be related to #187636, since these builds are running under Rosetta2 on aarch64-darwin?), Libepoxy is another common dependency failing due to a segmentation violation. So lots of issues to figure out, if anyone feels like they may be able to help, feel free to reach out.

This month marks an end to my Nix :heart: macOS sponsorship. Iā€™m grateful for having had the opportunity to focus on Nixpkgs development for macOS. Going forward I wonā€™t be able to spend as much time on this but I do want to finish the bump. If youā€™re short on time but want to help out the situation for Nix on macOS, consider contributing financially to the Open Collective.

Happy Holidays and a great 2023 to everyone!

20 Likes

Thank you for all the work youā€™ve done thus far! :heart:

3 Likes

Absolutely great, thanks so much for what you have done and the reports!

1 Like

Looks like you forgot to finish your sentence :slight_smile:

1 Like

I added a my contribution on open collective. Itā€™s not much but a little bit per month will surely help :heart:

6 Likes

June 2023

Holochain is graciously sponsoring improvements to the Darwin situation in Nixpkgs for six months. They leverage Nix for development environments and value being able to do so across both Linux and macOS.

@reckenrode has been doing great work on the stdenv with the goal of decoupling it from the bootstrap-tools. So far the LLVM version in the Darwin stdenv was tied to the version included in the bootstrap-tools. The rework of the stdenv makes it possible to bump the LLVM in the stdenv independently from the bootstrap-tools. This took a lot of work so I focused on unblocking it. The rework PR got merged in the meantime and more workā€™s underway to bump LLVM.

There was another discussion on how some maintainers experience Nixpkgsā€™ Darwin support as a burden. My main takeaway is that people want a team to contact for Darwin issues that block Linux channel progress.

Iā€™ve also resumed work on the stdenv bump. The source releases are a tangled mess of things (sometimes mutually) depending on each other. So we donā€™t want to bump individual source releases incrementally. But the full bump grew to over 130 commits (this is after some householding) and thatā€™s a bit rough for the review process. So Iā€™ve settled on a plan of introducing new versions of the source releases alongside the current ones. This way I can split the bump into small PRs focussing on individual releases and then when all of the releases have been merged, we can switch the entire set over all at once.

18 Likes

July 2023

The stdenv LLVM bump is progressing. Shout-outs to @reckenrode and everyone chipping in!

Iā€™ve continued splitting the SDK bump into smaller branches. One snag has been bumping the SDK itself, which dtrace depends on (in later versions we might be able to drop that dependency again), leading to a bit of a circular dependency situation because the SDK requires the stdenv and part of the stdenv requires the SDK.

12 Likes

September 2023

The stdenv LLVM bump is still in need of more reviews. @reckenrode is also working on fixing cross-compilation from aarch64-darwin to x86_64-darwin.

All the dependencies of the bootstrap-tools are split into branches. The other source releases should be buildable against the new bootstrap-tools and avoid a needless rebuild later. I have a patch for issue #150655, however, it makes the binutils build fail and thatā€™s pretty low down in the dependency chain and Iā€™ve been focusing on the prerequisite PRs for the LLVM bump instead. Iā€™m also trying to add the Compression framework to the SDKs, which is needed for newer system_cmds, but running into needing newer darwin-stubs. Either those need a new release or we can switch to fetching the repo instead.

9 Likes

A bit of extra context for the system_cmds update: what Iā€™m trying to do is see just how new we can go since the _cmds packages arenā€™t actually shipped with a particular SDK release. Iā€™ve got most of system_cmds 970.0.4 building, which is the release corresponding to macOS 14 (but using the default SDKs in nixpkgs). After that, I want to look at the other releases.

Iā€™m also replacing the build systems with Meson. Some of the packages try to use Appleā€™s Xcode projects while others have a custom build phase. Apple keeps changing the structure of things, and xcbuild is no longer maintained anyway, so I gave up on trying to make it work.

3 Likes

October 2023

The stdenv LLVM bump was merged. It introduced some issues but theyā€™re being addressed. The cross-compilation PR is still open, but, currently fixes required due to the bump are prioritized.

The patch for issue #150655 broke the binutils build because of unused argument warnings, which cause its configure script to misdetect the presence of standard library headers. Iā€™m still looking into the linker-side of things, the current patch only addresses the compiler-side. I will also continue with the Compression framework, to unblock newer system_cmds and then the stdenv work.

11 Likes

I would also like to mention two things: #265103 and overrideSDK.

The first PR will make CoreFoundation the default over the open-source CF on x86_64-darwin. It is planned for the next staging cycle. The change is being made due to compatibility issues on macOS 14 (possibly due to CorrFoundation hardening).

overrideSDK is a replacement for the apple_sdk_11_0.callPackage pattern. It takes a stdenv and an SDK version string, and it provides an adapted stdenv that uses frameworks from the requested SDK including propagated ones. It also ensures that a compatible xcbuild is used. overrideSDK can also be used to change the deployment target (in addition to the SDK version or just that).

overrideSDK is currently in staging-next and gaining features as needed. Once additional SDKs are introduced and the SDK hierarchy is standardized, it will be updated to support using SDK-specific libraries (such as simd).

10 Likes

The CoreFoundation PR was merged today as part of the merge of stage into staging-next (and the restart of builds). Most of the existing CF-centric stuff (hooks, etc) was left in place (in case the PR needs to be reverted but also to keep the change small), but I expect it will be cleaned up and removed for 24.05.

10 Likes

November 2023

Progress on the cc-wrapper fixes is slow, the large rebuilds make it hard to iterate. The Compression framework bump is currently stuck on a problem with the OpenDirectory framework.

Iā€™ve opened an overview issue for the first source release bumps. The intent is to introduce the new source releases alongside the old ones to enable review in small units and then switch over when everythingā€™s merged. Iā€™ve left the PRs in draft for now in case we can come up with a better way than the nested attribute set in the source releases expression.

5 Likes

Have you considered using ccache?

Is it possible to open a draft PR to start getting feedback and enable others to be able to help build and test?

Also, FYI, thereā€™s also another attempt at fixing it at: cc-wrapper: Enter C++ mode when -lc++ is passed by judofyr Ā· Pull Request #269694 Ā· NixOS/nixpkgs Ā· GitHub.