On the future of Darwin SDKs (or how you can stop worrying and put the SDK in build inputs)

reckenrode · August 14, 2024, 4:32am

Since I’ve mentioned it in a response to a few PRs and have been talking about it for the last six or so months on Matrix, I thought I would post an update to explain a bit what is coming for Darwin SDKs.

Disclaimer: This is a work in progress. Things could change, but it’s shaping up to be pretty good.

Background

There are currently three SDKs available for Darwin in nixpkgs: 10.12, 11.0, and 12.3. The default SDK is different between platforms, and there are several different ways of using SDKs. It is possible to cross-compile from x86_64-darwin to aarch64-darwin but not the other way around. The SDKs are built with different patterns, and updating them is difficult. There are also two ways of overriding the default SDK, and the better one is not even documented (which is my fault).

The current situation works and has served nixpkgs for many years, but it can be improved. Instead of packaging frameworks individually, which contributes to making it difficult to update the SDK and makes overriding SDKs impossible to do generally (because dependencies can be propagated in ways that are invisible to Nix, the language), the SDK can be provided as a single package.

The Solution

The solution is to add another pattern. See also: xkcd: Standards.

More seriously, yes. Native tooling on Darwin expects there to be an SDK at a standard location. It can be changed using xcode-select, but the location can also be specified via SDKROOT and -isysroot. The change is to take advantage of these to implement support for a single SDK package. Hooks are also used to make sure that only one SDK can be active at a time (though technically SDKROOT_FOR_BUILD and SDKROOT_FOR_TARGET will also be supported).

By default, the stdenv will come both a default SDK (same as current) as xcrun. To change the SDK used in your package, you add it to your package’s buildInputs. The hooks will make sure that’s what you get (unless a dependency propagates a newer SDK, then you will use that). For an example of what this looks like, see this diff to psutil: psutil with the new pattern · GitHub.

I have been prototyping the new SDK pattern using a stdenv adapter. I have successfully built Wine and MoltenVK as well as pytorch with MPS support.

Supported SDKs

The following SDKs will be supported: 10.12.2, 10.13.2, 10.14.6, 10.15.6, 11.3, 12.3, 13.3, 14.4. When macOS 15 is released, the 15.0 SDK will be added once it is available.

Source releases that are built will be updated as new ones are made available. Which ones in particular and their versions will be included in the release announcement. Most of the _cmds packages will still be built as will libiconv, libresolv, and libutil. OpenBSM is also included but technically a third party project.

Also, Cross-compilation

The new SDK pattern is being written with cross-compilation in mind. Darwin to Darwin will be supported from day 1 when it is merged. I am modifying cc-wrapper.sh to take DEVELOPER_DIR and SDKROOT into account (including the different role variants). Also, because the stdenv is being reworked to use the LLVM bootstrap to build, cross-compilation should stay working because anything that breaks it should also break the regular bootstrap.

Current Status

I’ve successfully completed the prototype. As mentioned above, I’ve been able to build a few packages with it. I’m currently working on building the bootstrap with the new SDKs.

Once the SDKs successfully bootstrap, I have some more work to do. I need to finish up the cross-compilation support. I also need to test the channel blockers and that packages with known issues (such as Wezterm) are able to build. I also need to update overrideSDK to work with the new SDKs.

Oh, and write documentation. Being able to change the SDK on a package by putting it in the package’s build inputs feels like magic. The documentation is going to explain how to do that, change the deployment target, and what’s actually included (such as propagated packages).

Transition to the New Pattern

At the soonest, maybe the next staging-next cycle. It may be the one after that. This will definitely be in 24.11. The initial PR will only add the pattern and update the bootstrap (and other things as necessary). The frameworks will remain, but they will be stubs that do nothing. This is to avoid breaking existing packages.

Once the frameworks have been removed from packages in nixpkgs, they will be added to darwin-aliases.nix. This could happen for 24.11, but it may happen afterward. For the next release, a warning will be added. The soonest the frameworks would be removed is 26.05. This is to give people enough time to update.

szlend · August 14, 2024, 7:58am

Love reading through your updates and it makes me really excited about the future of Darwin in nixpkgs! Thank you for all your hard work!

reckenrode · August 21, 2024, 12:41pm

Status update: I have successfully bootstrapped the stdenv on both aarch64-darwin and x86_64-darwin.

$ nix build .#legacyPackages.{aarch64,x86_64}-darwin.stdenv 
$ nix-store -qR ./result{,-1} | rg apple-sdk
/nix/store/idl7zza4f2wx1cvyz88m88vjwpqwrb4b-apple-sdk-11.3
/nix/store/jc7hi9v1hzmzp38l9868as5ss2znkh85-apple-sdk-10.12.2

Next steps: I am currently attempting to build Wezterm on x86_64-darwin. It is a package with known issues (see: Build failure: Wezterm (x86-64 darwin) · Issue #239384 · NixOS/nixpkgs · GitHub) that should serve as a good test case for the new pattern.

It’s getting closer to when I will be able to open the PR, but there is still work to be done. I need to find a source of locale data, update the other _cmds packages, fix xcrun -find, create a proxy ICU package, and more.

reckenrode · August 21, 2024, 12:48pm

Fun fact: The stdenv bootstrap depends on the following SDKs: 10.12 (x86_64 only), 10.13 (x86_64 only), 11.3, 12.3, and 14.4. The deployment target is still the current default (10.12 and 11.0 for x86_64- and aarch64-darwin respectively).

khaneliman · August 23, 2024, 5:04am

Thanks for all your hard work on improving the state of Darwin support in nixpkgs. I enjoy seeing your progress and watching your updates through Matrix and it’s very exciting how close “The Great Refactoring” is getting to come to fruition!

reckenrode · September 1, 2024, 3:00pm

Current status: I’m working on building the Darwin channel blockers and my configs.

Currently some cross-compiled stuff builds. I’m able to cross-compile cctools, but compiler-rt fails to build (due to a mistake I made in the updated derivation).

Most languages build. I spent most of last week working on Swift, which had some issues with the new SDK pattern. It now builds and propagates the 13.3 SDK (to match the Swift version to the upstream SDK version) and propagates a deployment target of 10.15 (or 11.0 on aarch64-darwin). I’m currently working on changes needed for GHC to build.

There have been a few changes to the approach.

libunwind was propagated by the SDK. It is now propagated by the compiler-wrapper. Darwin is as close to being useLLVM as it can be without using LLD.
OpenBSM is no longer propagated as part of the SDK. It does not contain headers included in newer SDKs, which breaks clang’s module definitions (needed by Swift).
cctools now has a libtool output for packages that need just libtool. Those that are creating a derivation just for libtool can switch to using cctools.libtool. However,
Since the SDK includes xcrun, libtool can be included in the SDK’s toolchain without being in PATH. libtool (from cctools) can be found with xcrun --find libtool and used with xcrun --run libtool.
CMake has been updated to properly find libraries in the SDK. It was trying to search for them in $SDKROOT/var/empty/lib instead of $SDKROOT/usr/lib. This was needed for the find_library(MATH_LIBRARY m) pattern used to find libm to work.

On breaking changes: The refactor tries not to break things, but it prioritizes the proper functioning of the SDK pattern over trying to be too compatible (e.g., to avoid unexpected SDK interactions). These are some known breaking changes:

Packages that make assumption about how the SDK is built and laid out will probably break. The old packages will remain available as stubs, but all they contain is a README telling you to switch to the SDK. If you assume that frameworks or libraries are found at certain paths in the framework packages, you should try to rely on them being set up to be found normally or use $SDKROOT/usr/lib.
Darwin’s libc stubs and headers are located dynamically at build time. The wrappers normally take care of this, but if you need to link libc manually, use $SDKROOT/usr/lib and $SDKROOT/usr/include instead of hardcoding a path to libc. So far, this has only affected libredirect, but it could affect other packages that build and link without wrappers.
Possibly, but hopefully not breaking: xcbuild no longer uses wrappers/shims. xcodebuild and xcrun are the full binaries. xcrun checks its toolchain (currently only containing libtool) then PATH to locate binaries. xcodebuild uses SDKROOT, which is provided by the stdenv pointing to the nixpkgs SDK.

Other news: @emily opened a PR to add an announcement to the release notes for 24.11 that it will be the last version to use 10.12 as the default SDK and deployment target on x86_64-darwin. The default SDK version will be updated for 24.05 (probably to the 11.3 SDK with a 11.0 deployment target). This will align x86_64-darwin with aarch64-darwin.

github.com/NixOS/nixpkgs

nixos/doc/rl-2411: warn about upcoming macOS version requirement

NixOS:master ← emilazy:push-xnqswoxruuln

opened 11:32PM - 31 Aug 24 UTC

emilazy

+6 -0

## Description of changes For a long time now, the SDK and minimum target ver…sion for `x86_64-darwin` has been stuck on macOS 10.12. In the past, the minimum SDK was updated quite regularly; at first, the current situation was just because updating the SDKs was excessively burdensome and nobody was up for doing the work, but the introduction of `aarch64-darwin` with its macOS 11 default SDK has resulted in a long‐term fracture of the two platforms. Per <https://endoflife.date/macos>, macOS 10.12 has not received an update since 2017 and went out of security support 5 years ago. Trying to support it in Nixpkgs has been a large burden on the Darwin maintainers, resulting in workarounds, porting work, and even patching functionality out of applications. The existence of Nix users using a macOS version this old is, to my knowledge, entirely theoretical, and we pay in both maintenance costs and functionality: for instance, applications built for `x86_64-darwin` do not support automatic dark mode switching by default. This situation has always been suboptimal, but it is now becoming untenable. Python, a critical component of the Nixpkgs standard environment for builds, is dropping support for versions older than 10.13 in 3.13: <https://www.python.org/downloads/release/python-3130rc1/>. Qt 6 only supports macOS 11 and newer. libuv only supports the versions Apple does, and is a ticking time bomb due to its use in the standard environment. QEMU only supports the last two macOS releases, and won’t build with an SDK older than macOS 12; we previously vendored a set of backporting changes and functionality‐removing reverts to keep it building for 10.12, but this also became overly onerous, and we gave up in <https://github.com/NixOS/nixpkgs/pull/338598>. `x86_64-darwin` is a platform with a limited upstream future. Apple no longer sells any hardware that runs it natively, and it is unclear how much longer they will support it in the operating system. There are still many users of the platform, myself included, so we shouldn’t drop support for it prematurely, but it’s unreasonable to try and patch the entire world to keep it supporting insecure versions of the OS that only run on hardware that is no longer sold. Therefore, this adds a release note to warn users ahead of time that 25.05 will only support macOS 11 and newer, as suggested by the 24.05 release team when the possibility of bumping the required version was raised. Why target Big Sur, rather than any other version? The reason is simple: it’s the same SDK and deployment target as `aarch64-darwin`. There are many packages that work on `aarch64-darwin` but not `x86_64-darwin`, and Darwin maintainers frequently need to be called in to fix things that work fine on the newer platform but not the older one. This change will increase the health of `x86_64-darwin` by aligning the SDK versions and support between the two platforms; the vast majority of packages that work on one will Just Work on the other. macOS 11 is almost four years old and has itself been out of security support for a year now, but as the first version to support Apple Silicon, it’s a far more compatible base for us to build our Darwin packages for. Any future change in supported versions should be synchronized between the two Darwin architectures. When 25.05 is released, users on old, unsupported versions of macOS will have the following options: * Update to a new macOS version. For users that are on hardware that Apple has dropped support for, OpenCore Legacy Patcher (<https://dortania.github.io/OpenCore-Legacy-Patcher/>) can enable the use of newer macOS versions on hardware even older than 10.12 supports. * Install NixOS. That obviously precludes the use of macOS software (though most of that software has already dropped support for 10.12), but will give users a secure, supported operating system that we can actually own the support for going forward. * Keep using 24.11 forever. Since they’re not getting updates to their OS and core applications anyway, this is likely to be acceptable to many users. * Switch to MacPorts. They support all the way back to 10.6 for `x86_64-darwin` by building packages separately for every OS release, though not every package is available for every version. * Send patches. We *may* accept non‐invasive patches to keep certain critical packages (such as the core `stdenv` packages) building for old OS versions, on a case‐by‐case basis, but we can’t guarantee it. This will ultimately have to be a decision made by package maintainers and personally I doubt this will be a viable path to sustainably support older versions. cc @NixOS/darwin-maintainers ## Things done - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [ ] x86_64-darwin - [ ] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [ ] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc

reckenrode · September 8, 2024, 2:37am

Quick status update: My configs build. I’m able to cross-compile stuff. I’ve made a few fixes and simplifications. Overrides are gone. The SDK no longer needs to replace the availability headers and uses Csu 88 on x86_64-darwin with all SDKs.

Next steps: Once my configs build again after the recent changes, I’ll kick off a build of my regression flake. After all that builds, it’s time to clean up my commit history, rebase on staging, and prepare the PR.

Stdenv Bootstrap Stats: 24.05 builds Python five times during the stdenv bootstrap. Unstable builds it twice. The refactor builds it once. The new stdenv bootstrap does a much better job of not building more than it needs to build.

Branch	aarch64-darwin	x86_64-darwin
24.05	1196	1296
Unstable	1083	1205
Darwin refactor	850	866

Source: nix-store -qR $(nix-instantiate . -A stdenv) | wc -l

reckenrode · September 17, 2024, 4:01am

Today’s status: I finished packaging the updates source releases. The regression flake is looking pretty good. There are a few failures I still need to investigate (such as source-built .NET on x86_64-darwin and libcurl static), but it’s looking really good overall, and I wouldn’t consider those blockers.

I also spent some time tonight writing the script to parse Software Update catalogs and display the Xcode version associated with the available SDKs, which should make picking the right one easier for future updates. I tested it by adding the 15.0 SDK, though the plan is for someone else to do the update to test the process, so the 15.0 SDK will not be included in my PR.

I tried building MoltenVK, but it failed due to availability limitations in clang 16. The issue is that clang 17 automatically defines __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__, which the 14.4 and 15.0 SDK require for availability checks to function properly. Once I switched to clang 17, MoltenVK built with the 15.0 SDK.

I also want to investigate building ICU using the packaging in nixpkgs instead of Apple’s makefile. That should make it easier to implement static builds and also provide the expected support files (e.g., for pkg-config). For 25.05, I want to explore making additional source releases the default on Darwin (such as ICU and libpcap).

Next steps: Investigate the last few failures then start working on a final bootstrap before I start cleaning up my branch in preparation for the PR. The final bootstrap includes some fixes I have been deferring as well as backporting llvm/llvm-project@c8e2dd8 to earlier versions of clang.

In the unlikely event 15.0 source releases are made before I can open the PR, I’ll do a spot check to see if everything builds without issue. If so, they would be included. Otherwise, the 15.0 source releases will be included in 25.05.

Source Release Versions

The source releases correspond to those shipped in 14.6. When possible, they are built with the default SDK. A handful (network_cmds, top) require 11.0. Note that libsbuf is not in the list. It ought to be a source release, but one has not been made. I ported it from FreeBSD 14.1 to match the ABI on Darwin and API provided in the SDK.

AvailabilityVersions - 141.1
Csu - 88
ICU - 74000.403
PowerManagement - 1630.120.8
adv_cmds - 216.100.1
basic_cmds - 67.100.1
bootstrap_cmds - 133.140.1
copyfile - 196.120.5
developer_cmds - 79
diskdev_cmds - 718.140.2
doc_cmds - 66
file_cmds - 430.140.2
libiconv - 102
libpcap - 131
libresolv - 79
libutil - 72
mail_cmds - 38.0.1
misc_cmds - 44
network_cmds - 671.120.2
patch_cmds - 61
remote_cmds - 303.141.1
removefile - 70.100.4
shell_cmds - 309.120.3
system_cmds - 979.120.4
text_cmds - 165.100.8
top - 137.100.2

Note: All of these build binaries, including several that previously only provided headers (such as copyfile, removefile, and libutil). It took some work to port them, but they should allow newer APIs to be used even on older systems.

reckenrode · September 29, 2024, 1:38am

I haven’t attempted the backport yet. I’m going to do that on my next rebuild (probably while rebasing on staging after switching my repo to Jujutsu). I can confirm that I was able to base the ICU derivation on the top-level derivation.

ICU now builds the standard dylibs plus headers (and pkg-config files, etc). I reexport those dylibs in libicucore.dylib. I can confirm it has a compatible ABI because I was able to build .NET 8 from source, which requires it. It can also be built statically.

I fixed the xcbuild infinite recursion (when using xcrun with system paths, it would find /usr/bin/<stub>, which would try to execute xcrun, which would find the stub, etc). I also confirmed that Qt WebEngine still builds after cleaning up the binaries I propagate.

Finally, I successfully built all of the source releases including cross-compilation and static versions except for bootstrap_cmds and those that depend on Perl (because the version of Perl in my branch currently is old and broken when cross-compiling).

I switched my nixpkgs repo to Jujutsu, so I can clean up the history to prepare it for a draft PR. While people are reviewing the draft, I can finish up the last few pieces. I’m also starting to split off some of the package fixes into separate branches (which Jujutsu handles very nicely), so the PR can be focused as much as possible on the Darwin stuff.

Source Release Updates

If Apple happens to release the macOS 15 source releases before the PR, I will check to see if they all still build. If they do, the PR will include the macOS 15 source releases instead of the macOS 14.6 ones. If not, I’ll push them to 25.05.

I also added IOKitTools 124 to the source releases. It’s needed by slackdump: init at 2.5.11 by Bot-wxt1221 · Pull Request #343156 · NixOS/nixpkgs · GitHub.

viraptor · September 29, 2024, 9:21am

If they do, the PR will include the macOS 15 source releases instead of the macOS 14.6 ones. If not, I’ll push them to 25.05.

Isn’t this purely an addition, i.e. possible to backport safely anyway?

reckenrode · September 29, 2024, 1:28pm

It depends on the scope of what’s required to make them build. The current default SDK on x86_64-darwin is 10.12, but it’s being bumped to 11.0 for 25.05. If any of the updated source releases start requiring 11.x APIs, additional work would be required to backport them.

I’ve hesitated to link it here because the branch should be considered unstable (as in commit ids change almost every push), but these are the source release packages.

Some of the workarounds have required providing replacement implementations of newer APIs, using older versions of some files, adding availability checks, etc. Once both Darwin architectures default to the 11.x SDK, I’d like to drop the compatibility patches for maintainability reasons (to avoid bitrot since they’re mostly conditioned on SDK version).

reckenrode · September 29, 2024, 1:35pm

I should add that network_cmds is its own kind of horrible. It uses a lot of private ioctls. One of the bad things about using kernel headers is they are infectious. If you include them directly, you usually have to include even more kernel headers, and they inevitably end up conflicting with headers from the SDK. I’ve got a pattern for dealing with it, but it’s ugly.

I tried building network_cmds for 10.12, but there were too many errors, and I’d already spent a couple of days making it build for 11.0, so I gave up on it. It’s easily the worst of the source releases to make build, and I include a couple where I had to reverse engineer unavailable/undocumented, private API headers to get them to build.

(Additionally, netstat really requires 13.0 or newer. I ended up copying the files from an older version, so it can target 11.0. I have some ideas for falling back for older releases at runtime, so users on newer releases get the extra functionality, but the work is non-trivial and would have to be post-24.11.)

reckenrode · October 2, 2024, 1:01pm

Quick status update: I’m mostly finished with cleaning up my history. I rebased onto staging from last night, and now I’m working through cleaning up the conflicts now.

It seems likely that I should be able to open a draft PR sometime this evening.

Source Release Updates

Apple released the macOS 15 source releases. I updated the source releases to the versions from macOS 15. Once I finish rebasing, I’ll also update xar and add xarMinimal for the stdenv bootstrap.

reckenrode · October 3, 2024, 1:54am

I have created the draft PR.

github.com/NixOS/nixpkgs

darwin: change the SDK pattern, update the SDKs, and update source releases

NixOS:staging ← reckenrode:darwin-sdk-refactor

opened 01:51AM - 03 Oct 24 UTC

reckenrode

+12134 -18096

There are currently three different SDKs in nixpkgs, which are all built slightl…y differently. The 10.12 SDK is built from a mixture of source releases, stubs and headers (from source releases), and headers from the upstream SDK. The 11.0 (really, 11.1) SDK is built mostly from the upstream SDK. These two SDKs have served nixpkgs well for several years, but they have several limitations. The 12.3 SDK was added several months ago based on a stripped down version of the pattern used in https://github.com/NixOS/nixpkgs/pull/229210. It is a newer pattern still, but it shares many of the same limitations. * Adding a new SDK version is a non-trivial amount of work. After finding the right SDK package source, you have to (attempt to) generate a dependencies file to make sure that adding a particular framework also propagates the frameworks it needs. Sometimes these frameworks cannot be determined automatically by some other means (such as trial-and-error). * The newer SDKs don’t make use of the source releases much at all except for applications (various cmds packages), libiconv, and ICU (for .NET, mostly). * Knowing which frameworks to use is not obvious, especially if you do not have access to Darwin hardware. Packages that could be built for Darwin sometimes are not because the maintainers don’t want to deal with trying different things until ofborg successfully builds the package, assuming they know enough to know which frameworks even to use. * Overriding the SDK cannot be done generally. Some packages require newer APIs, but it is not possible to use a newer SDK by default. Some packages cannot be built with a newer SDK (e.g., PyTorch 2.3 and Qt WebEngine 5.15 cannot built with any SDK newer than the 13.3 SDK). Some packages perform static checks and use newer SDK features unconditionally if detected (e.g., SciPy’s Accelerate-based blas support). Consequently, an SDK corresponding to the minimum supported version is used with the ability to override the SDK (e.g., with `overrideSDK`). To elaborate on the above, `overrideSDK` works currently by enumerating the transitive, propagated inputs of a package. This set of packages is then examined, and propagated SDK inputs are replaced with versions corresponding to the requested SDK. The problem is that the set of build inputs does not and cannot include all possible dependencies because dependencies can be propagated in other ways. Packages can capture a framework and include it in their build files (such as abseil-cpp prior to https://github.com/NixOS/nixpkgs/pull/265395 or `rustPlatform`, which captures various build/host/target clangs in its hooks. The solution is to stop trying to fight how the platform works and embrace it: make the SDK a package laid out like a normal SDK would be on Darwin. The tooling expects it. Instead of having to hack build systems to deal with nixpkgs idiosyncrasies, most builds should be able to Just Work™ by default. --- This PR adds a new SDK pattern for Darwin that provides a single SDK package for each version. A default SDK is provided as an `extraBuildInput` in the Darwin stdenv. To use a newer SDK version, add it as a `buildInput`. A hook is used to pick which SDK of those available is used (currently always the newest); and it is possible to have a different SDK for build, for host, and for target platforms. Most importantly, it is _never_ possible to have more than one SDK active. Weird errors due to semicolons or other punctuation should be a thing of the past. Additionally, a separate hook is also provided to set the minimum version (also known as the deployment target). If your package specifies that it works on 11.0 and newer, you can use the hook to do that. If your package is a library, you can propagate the hook to make sure users of your library also target the correct version. This combination allows SDKs and SDK requirements to be propagated to packages without using a new scope and `callPackage`. Packages that would be prevented from using `by-name` can now be moved there and use whatever SDKs they need or are propagated to them by their dependencies. To provide better compatibility with build systems that assume Xcode is available, `xcrun` is also included in the stdenv. It and the xcbuild derivation have been reworked to find the SDK dynamically. It should hopefully no longer be necessary to patch out invocations of xcrun in build systems (e.g., `xcrun --find clang` in a build will return clang from nixpkgs). In addition to adding the new SDK pattern, this PR fixes cross-compilation on Darwin. It should now be possible to build static packages for both aarch64- _and_ x86_64-darwin (assuming the packages support it). It should also be possible to cross-compile between both architectures (again assuming the packages support it). ## New SDKs The following SDKs are added in this PR: 10.12.2, 10.13.2, 10.14.6, 10.15.6, 11.3, 12.3, 13.3, and 14.4. The 15.0 SDK has been left out to be added later as a test of the SDK update infrastructure to assess and confirm how easy it is to add a new one. In my testing, adding the 15.0 SDK is a ~5 minute task. Hopefully, it goes as easily for @emilazy. ## Source Releases The 10.12 SDK builds several frameworks and libraries from source. It also makes use of their headers. The new SDK pattern no longer tries to get headers from the source releases. They’re just too inconsistent with the headers in the SDK. Some headers are even older than the 10.12 because newer sources are not available or complete enough to use. However, this update actually builds and uses _more_ source releases than before. The following libraries are built. Of these, libresolv is new on aarch64-darwin, libsbuf is new for both (ported from FreeBSD due to the lack of a source release), libpcap is new for both, and copyfile and removefile are built from source for the first time. * ICU * libiconv * libresolv * libpcap * libsbuf * libutil * copyfile * removefile * xar Currently, libiconv and libresolv are propagated with the SDK on both platforms. For 25.05, libpcap and ICU may be made to be the default implementation on Darwin due to containing Apple-specific changes. It is also worth noting: all source releases correspond to the source released for macOS 15. When possible, it has been backported to build with the default SDKs. ### Source release version updates * AvailabilityVersions: 140.1 -> 143.3 * ICU: 66108 -> 74221 * IOKitTools: init at 125 * PowerManagement: 572.50.1 -> 1740.0.7 * adv_cmds: 163 -> 231 * basic_cmds: 55 -> 70 * bootstrap_cmds: 121 -> 136 * copyfile: 138 -> 213 * developer_cmds: 62 -> 79 * diskdev_cmds: 593 -> 735 * doc_cmds: init at 66 * file_cmds: 264.1.1 -> 448.0.3 * libiconv: 99 -> 107 * libpcap: init at 135 * libresolv: 64 -> 83 * libutil: 47.30.1 -> 72 * mail_cmds: init at 38.0.1 * misc_cmds: init at 44 * network_cmds: 606.40.2 -> 696 * patch_cmds: init at 62 * remote_cmds: init at 303.0.2 * removefile: 45 -> 75 * shell_cmds: 187 -> 319.0.1 * system_cmds: 970.0.4 -> 1012 * text_cmds: 99 -> 190.0.1 * top: 108 -> 139 * xattr: replace with xattr from file_cmds * xar: 498 -> 501 ## Transition Strategy Because this is a non-trivial change to how Darwin works, a transition is necessary. This update tries to avoid breaking existing packages, but there are breaking changes (see below). To minimize breakage, all existing SDK frameworks and libraries as well as dropped source releases (anything not above) have been replaced with stubs. These stubs contain a `README` advising the user to read the documentation on how to update to the new pattern. Currently, these stubs do nothing other than provide stubs. After a treewide has been done, they will moved to `darwin-aliases.nix`. It is expected that these aliases will remain in place until at least the 26.05 release. ### Breaking Changes * libiconv is no longer linked automatically. Since it is propagated by the SDK by default, including this hook would link libiconv to everything, which may not be desirable. Failing to link libiconv is a build system error. The upstream builds should be fixed to correct the mistake. * CoreFoundation is no longer linked automatically. Same as above. This is a build system issue. Hooks really shouldn’t be used to work around mistakes upstream, especially when the issues are not pervasive. * Anything that depended on the layout of the old frameworks will need to be updated to work with the new pattern: * Frameworks can be found at `SDKROOT/System/Library/Frameworks`; * The SDKROOT for a particular SDK package is available using the `sdkroot` attribute on the package (e.g., `apple-sdk.sdkroot`). * Darwin is now a “system” libc++ platform. LLVM will now always use the libc++ built in the stdenv unless you use `llvmPackages.libcxxStdenv`. This aligns Darwin with Linux where `llvmPackages.stdenv` uses the default libstdc++. * Darwin’s libc is determined dynamically. `darwin.libSystem` contains empty `lib` and `include` folders to avoid compiler and linker warnings that may be turned into errors. The libSystem stubs can be located at `$SDKROOT/usr/lib/libSystem.tbd`. * Packages may try to build additional functionality due to the presence of SDK frameworks. For example, Python no longer supports the `configd` option to disable the `_scproxy` module. The SDK always has the SystemConfiguration framework. * Vendored packages have been removed from the SDK. Between all the different versions, more than 50 packages are vendored. Those that are conventionally added as inputs are excluded from the SDK. Just add them as inputs like normal. Those that are part of the SDK are propagated. This includes libiconv and libresolv as well as the headers for ncurses (which are needed by Swift). Note that libutil is not propagated because its headers may cause problems with some packages (particularly Python). If you need libutil, add `darwin.libutil` as a build input. * Old source releases are no longer available, but their source for the SDK is accessible via the `apple-sdk.sourceRelease` function. This is mostly useful for copying private headers for use to build a derivation. See various source releases for examples. ## Closed Issues and PRs - closes #86299 - closes #101229 - closes #229210 - closes #242666 - closes #264091 - closes #269646 - closes #278348 - closes #273442 - closes #284629 - closes #308366 ## Things Still to be Done This PR is almost completely ready. If need be, it could be merged into staging to make the next staging-next cycle. The following. - [ ] Rework locale data. In a pinch, I could duplicate the current logic. Apple appears to be using the same source of data as FreeBSD, but the data from the FreeBSD doesn’t actually work with their tools. Honestly, I’m not sure setting the environment variable even works. - [ ] `netstat` doesn’t print active Internet connections. It prints other kinds but not those. It requires further investigation. - [ ] Bootstrap tools need updated. This could wait until after this is merged, but it will need to be done before this hits master to prevent the channel from being blocked. ## Related PRs These issues were found during testing. This [flake](https://gist.github.com/reckenrode/58933e82c687d8026fe19630c0ed699f) was used to help find issues. Some of these can target staging or even master without waiting. Others depend on this PR or even other PRs. (list will be updated as open PRs.) - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [x] x86_64-darwin - [x] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [x] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc

reckenrode · October 11, 2024, 12:12am

The PR has been merged into staging. Follow ups are being reviewed and merged to fix breakage and update some packages. You can see the improvements it allows in the following PRs (particularly Qt):

github.com/NixOS/nixpkgs

moltenvk: update for the new xcbuild, the new SDK, and the new version plus cleanup

NixOS:staging ← reckenrode:push-xrxtzvrtlpsl

opened 11:29PM - 06 Oct 24 UTC

reckenrode

+39 -45

This PR cleans up MoltenVK and moves it to `pkgs/by-name` in addition to updatin…g it for the new SDK pattern in https://github.com/NixOS/nixpkgs/pull/346043. ## Things done - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [x] x86_64-darwin - [x] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [ ] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc

github.com/NixOS/nixpkgs

qt{5,6}: update for the new Darwin SDK

NixOS:staging ← reckenrode:push-nyvqzpppwmom

opened 02:54AM - 08 Oct 24 UTC

reckenrode

+66 -308

Qt 5.15 needs to know the exact location of the OpenGL framework, so it needs up…dated for the new SDK pattern. I also updated Qt 6. Both are examples of how the new Darwin SDK can simplify derivations and improve support (e.g., Qt WebEngine builds again on Darwin). ## Things done - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [x] x86_64-darwin - [x] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [ ] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc

reckenrode · October 11, 2024, 1:09pm

@emily added the 15.0 SDK in apple-sdk_15: init at 15.0 by emilazy · Pull Request #347862 · NixOS/nixpkgs · GitHub. An SDK for every major version from 10.12 up to (and including) 15.0 is now available in nixpkgs. I have a few follow ups still left to do (writing documentation, fixing MacVim, etc), but the new SDK pattern is available on staging now.

PhilipTaron · October 12, 2024, 12:22am

This is incredibly powerful work, Randy. Kudos to you and the whole NixOS on macOS team for demonstrating how to cut through incredibly tricky technical knots without causing the ecosystem to fall off a cliff. I’m in awe of your ability to absorb the complexity without losing yourself in it.