On the future of Darwin SDKs (or how you can stop worrying and put the SDK in build inputs)

reckenrode · August 14, 2024, 4:32am

Since I’ve mentioned it in a response to a few PRs and have been talking about it for the last six or so months on Matrix, I thought I would post an update to explain a bit what is coming for Darwin SDKs.

Disclaimer: This is a work in progress. Things could change, but it’s shaping up to be pretty good.

Background

There are currently three SDKs available for Darwin in nixpkgs: 10.12, 11.0, and 12.3. The default SDK is different between platforms, and there are several different ways of using SDKs. It is possible to cross-compile from x86_64-darwin to aarch64-darwin but not the other way around. The SDKs are built with different patterns, and updating them is difficult. There are also two ways of overriding the default SDK, and the better one is not even documented (which is my fault).

The current situation works and has served nixpkgs for many years, but it can be improved. Instead of packaging frameworks individually, which contributes to making it difficult to update the SDK and makes overriding SDKs impossible to do generally (because dependencies can be propagated in ways that are invisible to Nix, the language), the SDK can be provided as a single package.

The Solution

The solution is to add another pattern. See also: xkcd: Standards.

More seriously, yes. Native tooling on Darwin expects there to be an SDK at a standard location. It can be changed using xcode-select, but the location can also be specified via SDKROOT and -isysroot. The change is to take advantage of these to implement support for a single SDK package. Hooks are also used to make sure that only one SDK can be active at a time (though technically SDKROOT_FOR_BUILD and SDKROOT_FOR_TARGET will also be supported).

By default, the stdenv will come both a default SDK (same as current) as xcrun. To change the SDK used in your package, you add it to your package’s buildInputs. The hooks will make sure that’s what you get (unless a dependency propagates a newer SDK, then you will use that). For an example of what this looks like, see this diff to psutil: psutil with the new pattern · GitHub.

I have been prototyping the new SDK pattern using a stdenv adapter. I have successfully built Wine and MoltenVK as well as pytorch with MPS support.

Supported SDKs

The following SDKs will be supported: 10.12.2, 10.13.2, 10.14.6, 10.15.6, 11.3, 12.3, 13.3, 14.4. When macOS 15 is released, the 15.0 SDK will be added once it is available.

Source releases that are built will be updated as new ones are made available. Which ones in particular and their versions will be included in the release announcement. Most of the _cmds packages will still be built as will libiconv, libresolv, and libutil. OpenBSM is also included but technically a third party project.

Also, Cross-compilation

The new SDK pattern is being written with cross-compilation in mind. Darwin to Darwin will be supported from day 1 when it is merged. I am modifying cc-wrapper.sh to take DEVELOPER_DIR and SDKROOT into account (including the different role variants). Also, because the stdenv is being reworked to use the LLVM bootstrap to build, cross-compilation should stay working because anything that breaks it should also break the regular bootstrap.

Current Status

I’ve successfully completed the prototype. As mentioned above, I’ve been able to build a few packages with it. I’m currently working on building the bootstrap with the new SDKs.

Once the SDKs successfully bootstrap, I have some more work to do. I need to finish up the cross-compilation support. I also need to test the channel blockers and that packages with known issues (such as Wezterm) are able to build. I also need to update overrideSDK to work with the new SDKs.

Oh, and write documentation. Being able to change the SDK on a package by putting it in the package’s build inputs feels like magic. The documentation is going to explain how to do that, change the deployment target, and what’s actually included (such as propagated packages).

Transition to the New Pattern

At the soonest, maybe the next staging-next cycle. It may be the one after that. This will definitely be in 24.11. The initial PR will only add the pattern and update the bootstrap (and other things as necessary). The frameworks will remain, but they will be stubs that do nothing. This is to avoid breaking existing packages.

Once the frameworks have been removed from packages in nixpkgs, they will be added to darwin-aliases.nix. This could happen for 24.11, but it may happen afterward. For the next release, a warning will be added. The soonest the frameworks would be removed is 26.05. This is to give people enough time to update.

szlend · August 14, 2024, 7:58am

Love reading through your updates and it makes me really excited about the future of Darwin in nixpkgs! Thank you for all your hard work!

reckenrode · August 21, 2024, 12:41pm

Status update: I have successfully bootstrapped the stdenv on both aarch64-darwin and x86_64-darwin.

$ nix build .#legacyPackages.{aarch64,x86_64}-darwin.stdenv 
$ nix-store -qR ./result{,-1} | rg apple-sdk
/nix/store/idl7zza4f2wx1cvyz88m88vjwpqwrb4b-apple-sdk-11.3
/nix/store/jc7hi9v1hzmzp38l9868as5ss2znkh85-apple-sdk-10.12.2

Next steps: I am currently attempting to build Wezterm on x86_64-darwin. It is a package with known issues (see: Build failure: Wezterm (x86-64 darwin) · Issue #239384 · NixOS/nixpkgs · GitHub) that should serve as a good test case for the new pattern.

It’s getting closer to when I will be able to open the PR, but there is still work to be done. I need to find a source of locale data, update the other _cmds packages, fix xcrun -find, create a proxy ICU package, and more.

reckenrode · August 21, 2024, 12:48pm

Fun fact: The stdenv bootstrap depends on the following SDKs: 10.12 (x86_64 only), 10.13 (x86_64 only), 11.3, 12.3, and 14.4. The deployment target is still the current default (10.12 and 11.0 for x86_64- and aarch64-darwin respectively).

khaneliman · August 23, 2024, 5:04am

Thanks for all your hard work on improving the state of Darwin support in nixpkgs. I enjoy seeing your progress and watching your updates through Matrix and it’s very exciting how close “The Great Refactoring” is getting to come to fruition!

reckenrode · September 1, 2024, 3:00pm

Current status: I’m working on building the Darwin channel blockers and my configs.

Currently some cross-compiled stuff builds. I’m able to cross-compile cctools, but compiler-rt fails to build (due to a mistake I made in the updated derivation).

Most languages build. I spent most of last week working on Swift, which had some issues with the new SDK pattern. It now builds and propagates the 13.3 SDK (to match the Swift version to the upstream SDK version) and propagates a deployment target of 10.15 (or 11.0 on aarch64-darwin). I’m currently working on changes needed for GHC to build.

There have been a few changes to the approach.

libunwind was propagated by the SDK. It is now propagated by the compiler-wrapper. Darwin is as close to being useLLVM as it can be without using LLD.
OpenBSM is no longer propagated as part of the SDK. It does not contain headers included in newer SDKs, which breaks clang’s module definitions (needed by Swift).
cctools now has a libtool output for packages that need just libtool. Those that are creating a derivation just for libtool can switch to using cctools.libtool. However,
Since the SDK includes xcrun, libtool can be included in the SDK’s toolchain without being in PATH. libtool (from cctools) can be found with xcrun --find libtool and used with xcrun --run libtool.
CMake has been updated to properly find libraries in the SDK. It was trying to search for them in $SDKROOT/var/empty/lib instead of $SDKROOT/usr/lib. This was needed for the find_library(MATH_LIBRARY m) pattern used to find libm to work.

On breaking changes: The refactor tries not to break things, but it prioritizes the proper functioning of the SDK pattern over trying to be too compatible (e.g., to avoid unexpected SDK interactions). These are some known breaking changes:

Packages that make assumption about how the SDK is built and laid out will probably break. The old packages will remain available as stubs, but all they contain is a README telling you to switch to the SDK. If you assume that frameworks or libraries are found at certain paths in the framework packages, you should try to rely on them being set up to be found normally or use $SDKROOT/usr/lib.
Darwin’s libc stubs and headers are located dynamically at build time. The wrappers normally take care of this, but if you need to link libc manually, use $SDKROOT/usr/lib and $SDKROOT/usr/include instead of hardcoding a path to libc. So far, this has only affected libredirect, but it could affect other packages that build and link without wrappers.
Possibly, but hopefully not breaking: xcbuild no longer uses wrappers/shims. xcodebuild and xcrun are the full binaries. xcrun checks its toolchain (currently only containing libtool) then PATH to locate binaries. xcodebuild uses SDKROOT, which is provided by the stdenv pointing to the nixpkgs SDK.

Other news: @emily opened a PR to add an announcement to the release notes for 24.11 that it will be the last version to use 10.12 as the default SDK and deployment target on x86_64-darwin. The default SDK version will be updated for 24.05 (probably to the 11.3 SDK with a 11.0 deployment target). This will align x86_64-darwin with aarch64-darwin.

github.com/NixOS/nixpkgs

nixos/doc/rl-2411: warn about upcoming macOS version requirement

NixOS:master ← emilazy:push-xnqswoxruuln

opened 11:32PM - 31 Aug 24 UTC

emilazy

+6 -0

## Description of changes For a long time now, the SDK and minimum target ver…sion for `x86_64-darwin` has been stuck on macOS 10.12. In the past, the minimum SDK was updated quite regularly; at first, the current situation was just because updating the SDKs was excessively burdensome and nobody was up for doing the work, but the introduction of `aarch64-darwin` with its macOS 11 default SDK has resulted in a long‐term fracture of the two platforms. Per <https://endoflife.date/macos>, macOS 10.12 has not received an update since 2017 and went out of security support 5 years ago. Trying to support it in Nixpkgs has been a large burden on the Darwin maintainers, resulting in workarounds, porting work, and even patching functionality out of applications. The existence of Nix users using a macOS version this old is, to my knowledge, entirely theoretical, and we pay in both maintenance costs and functionality: for instance, applications built for `x86_64-darwin` do not support automatic dark mode switching by default. This situation has always been suboptimal, but it is now becoming untenable. Python, a critical component of the Nixpkgs standard environment for builds, is dropping support for versions older than 10.13 in 3.13: <https://www.python.org/downloads/release/python-3130rc1/>. Qt 6 only supports macOS 11 and newer. libuv only supports the versions Apple does, and is a ticking time bomb due to its use in the standard environment. QEMU only supports the last two macOS releases, and won’t build with an SDK older than macOS 12; we previously vendored a set of backporting changes and functionality‐removing reverts to keep it building for 10.12, but this also became overly onerous, and we gave up in <https://github.com/NixOS/nixpkgs/pull/338598>. `x86_64-darwin` is a platform with a limited upstream future. Apple no longer sells any hardware that runs it natively, and it is unclear how much longer they will support it in the operating system. There are still many users of the platform, myself included, so we shouldn’t drop support for it prematurely, but it’s unreasonable to try and patch the entire world to keep it supporting insecure versions of the OS that only run on hardware that is no longer sold. Therefore, this adds a release note to warn users ahead of time that 25.05 will only support macOS 11 and newer, as suggested by the 24.05 release team when the possibility of bumping the required version was raised. Why target Big Sur, rather than any other version? The reason is simple: it’s the same SDK and deployment target as `aarch64-darwin`. There are many packages that work on `aarch64-darwin` but not `x86_64-darwin`, and Darwin maintainers frequently need to be called in to fix things that work fine on the newer platform but not the older one. This change will increase the health of `x86_64-darwin` by aligning the SDK versions and support between the two platforms; the vast majority of packages that work on one will Just Work on the other. macOS 11 is almost four years old and has itself been out of security support for a year now, but as the first version to support Apple Silicon, it’s a far more compatible base for us to build our Darwin packages for. Any future change in supported versions should be synchronized between the two Darwin architectures. When 25.05 is released, users on old, unsupported versions of macOS will have the following options: * Update to a new macOS version. For users that are on hardware that Apple has dropped support for, OpenCore Legacy Patcher (<https://dortania.github.io/OpenCore-Legacy-Patcher/>) can enable the use of newer macOS versions on hardware even older than 10.12 supports. * Install NixOS. That obviously precludes the use of macOS software (though most of that software has already dropped support for 10.12), but will give users a secure, supported operating system that we can actually own the support for going forward. * Keep using 24.11 forever. Since they’re not getting updates to their OS and core applications anyway, this is likely to be acceptable to many users. * Switch to MacPorts. They support all the way back to 10.6 for `x86_64-darwin` by building packages separately for every OS release, though not every package is available for every version. * Send patches. We *may* accept non‐invasive patches to keep certain critical packages (such as the core `stdenv` packages) building for old OS versions, on a case‐by‐case basis, but we can’t guarantee it. This will ultimately have to be a decision made by package maintainers and personally I doubt this will be a viable path to sustainably support older versions. cc @NixOS/darwin-maintainers ## Things done - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [ ] x86_64-darwin - [ ] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [ ] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc

reckenrode · September 8, 2024, 2:37am

Quick status update: My configs build. I’m able to cross-compile stuff. I’ve made a few fixes and simplifications. Overrides are gone. The SDK no longer needs to replace the availability headers and uses Csu 88 on x86_64-darwin with all SDKs.

Next steps: Once my configs build again after the recent changes, I’ll kick off a build of my regression flake. After all that builds, it’s time to clean up my commit history, rebase on staging, and prepare the PR.

Stdenv Bootstrap Stats: 24.05 builds Python five times during the stdenv bootstrap. Unstable builds it twice. The refactor builds it once. The new stdenv bootstrap does a much better job of not building more than it needs to build.

Branch	aarch64-darwin	x86_64-darwin
24.05	1196	1296
Unstable	1083	1205
Darwin refactor	850	866

Source: nix-store -qR $(nix-instantiate . -A stdenv) | wc -l

reckenrode · September 17, 2024, 4:01am

Today’s status: I finished packaging the updates source releases. The regression flake is looking pretty good. There are a few failures I still need to investigate (such as source-built .NET on x86_64-darwin and libcurl static), but it’s looking really good overall, and I wouldn’t consider those blockers.

I also spent some time tonight writing the script to parse Software Update catalogs and display the Xcode version associated with the available SDKs, which should make picking the right one easier for future updates. I tested it by adding the 15.0 SDK, though the plan is for someone else to do the update to test the process, so the 15.0 SDK will not be included in my PR.

I tried building MoltenVK, but it failed due to availability limitations in clang 16. The issue is that clang 17 automatically defines __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__, which the 14.4 and 15.0 SDK require for availability checks to function properly. Once I switched to clang 17, MoltenVK built with the 15.0 SDK.

I also want to investigate building ICU using the packaging in nixpkgs instead of Apple’s makefile. That should make it easier to implement static builds and also provide the expected support files (e.g., for pkg-config). For 25.05, I want to explore making additional source releases the default on Darwin (such as ICU and libpcap).

Next steps: Investigate the last few failures then start working on a final bootstrap before I start cleaning up my branch in preparation for the PR. The final bootstrap includes some fixes I have been deferring as well as backporting llvm/llvm-project@c8e2dd8 to earlier versions of clang.

In the unlikely event 15.0 source releases are made before I can open the PR, I’ll do a spot check to see if everything builds without issue. If so, they would be included. Otherwise, the 15.0 source releases will be included in 25.05.

Source Release Versions

The source releases correspond to those shipped in 14.6. When possible, they are built with the default SDK. A handful (network_cmds, top) require 11.0. Note that libsbuf is not in the list. It ought to be a source release, but one has not been made. I ported it from FreeBSD 14.1 to match the ABI on Darwin and API provided in the SDK.

AvailabilityVersions - 141.1
Csu - 88
ICU - 74000.403
PowerManagement - 1630.120.8
adv_cmds - 216.100.1
basic_cmds - 67.100.1
bootstrap_cmds - 133.140.1
copyfile - 196.120.5
developer_cmds - 79
diskdev_cmds - 718.140.2
doc_cmds - 66
file_cmds - 430.140.2
libiconv - 102
libpcap - 131
libresolv - 79
libutil - 72
mail_cmds - 38.0.1
misc_cmds - 44
network_cmds - 671.120.2
patch_cmds - 61
remote_cmds - 303.141.1
removefile - 70.100.4
shell_cmds - 309.120.3
system_cmds - 979.120.4
text_cmds - 165.100.8
top - 137.100.2

Note: All of these build binaries, including several that previously only provided headers (such as copyfile, removefile, and libutil). It took some work to port them, but they should allow newer APIs to be used even on older systems.