On the future of Darwin SDKs (or how you can stop worrying and put the SDK in build inputs)

reckenrode · October 3, 2024, 1:54am

I have created the draft PR.

github.com/NixOS/nixpkgs

darwin: change the SDK pattern, update the SDKs, and update source releases

NixOS:staging ← reckenrode:darwin-sdk-refactor

opened 01:51AM - 03 Oct 24 UTC

reckenrode

+12134 -18096

There are currently three different SDKs in nixpkgs, which are all built slightl…y differently. The 10.12 SDK is built from a mixture of source releases, stubs and headers (from source releases), and headers from the upstream SDK. The 11.0 (really, 11.1) SDK is built mostly from the upstream SDK. These two SDKs have served nixpkgs well for several years, but they have several limitations. The 12.3 SDK was added several months ago based on a stripped down version of the pattern used in https://github.com/NixOS/nixpkgs/pull/229210. It is a newer pattern still, but it shares many of the same limitations. * Adding a new SDK version is a non-trivial amount of work. After finding the right SDK package source, you have to (attempt to) generate a dependencies file to make sure that adding a particular framework also propagates the frameworks it needs. Sometimes these frameworks cannot be determined automatically by some other means (such as trial-and-error). * The newer SDKs don’t make use of the source releases much at all except for applications (various cmds packages), libiconv, and ICU (for .NET, mostly). * Knowing which frameworks to use is not obvious, especially if you do not have access to Darwin hardware. Packages that could be built for Darwin sometimes are not because the maintainers don’t want to deal with trying different things until ofborg successfully builds the package, assuming they know enough to know which frameworks even to use. * Overriding the SDK cannot be done generally. Some packages require newer APIs, but it is not possible to use a newer SDK by default. Some packages cannot be built with a newer SDK (e.g., PyTorch 2.3 and Qt WebEngine 5.15 cannot built with any SDK newer than the 13.3 SDK). Some packages perform static checks and use newer SDK features unconditionally if detected (e.g., SciPy’s Accelerate-based blas support). Consequently, an SDK corresponding to the minimum supported version is used with the ability to override the SDK (e.g., with `overrideSDK`). To elaborate on the above, `overrideSDK` works currently by enumerating the transitive, propagated inputs of a package. This set of packages is then examined, and propagated SDK inputs are replaced with versions corresponding to the requested SDK. The problem is that the set of build inputs does not and cannot include all possible dependencies because dependencies can be propagated in other ways. Packages can capture a framework and include it in their build files (such as abseil-cpp prior to https://github.com/NixOS/nixpkgs/pull/265395 or `rustPlatform`, which captures various build/host/target clangs in its hooks. The solution is to stop trying to fight how the platform works and embrace it: make the SDK a package laid out like a normal SDK would be on Darwin. The tooling expects it. Instead of having to hack build systems to deal with nixpkgs idiosyncrasies, most builds should be able to Just Work™ by default. --- This PR adds a new SDK pattern for Darwin that provides a single SDK package for each version. A default SDK is provided as an `extraBuildInput` in the Darwin stdenv. To use a newer SDK version, add it as a `buildInput`. A hook is used to pick which SDK of those available is used (currently always the newest); and it is possible to have a different SDK for build, for host, and for target platforms. Most importantly, it is _never_ possible to have more than one SDK active. Weird errors due to semicolons or other punctuation should be a thing of the past. Additionally, a separate hook is also provided to set the minimum version (also known as the deployment target). If your package specifies that it works on 11.0 and newer, you can use the hook to do that. If your package is a library, you can propagate the hook to make sure users of your library also target the correct version. This combination allows SDKs and SDK requirements to be propagated to packages without using a new scope and `callPackage`. Packages that would be prevented from using `by-name` can now be moved there and use whatever SDKs they need or are propagated to them by their dependencies. To provide better compatibility with build systems that assume Xcode is available, `xcrun` is also included in the stdenv. It and the xcbuild derivation have been reworked to find the SDK dynamically. It should hopefully no longer be necessary to patch out invocations of xcrun in build systems (e.g., `xcrun --find clang` in a build will return clang from nixpkgs). In addition to adding the new SDK pattern, this PR fixes cross-compilation on Darwin. It should now be possible to build static packages for both aarch64- _and_ x86_64-darwin (assuming the packages support it). It should also be possible to cross-compile between both architectures (again assuming the packages support it). ## New SDKs The following SDKs are added in this PR: 10.12.2, 10.13.2, 10.14.6, 10.15.6, 11.3, 12.3, 13.3, and 14.4. The 15.0 SDK has been left out to be added later as a test of the SDK update infrastructure to assess and confirm how easy it is to add a new one. In my testing, adding the 15.0 SDK is a ~5 minute task. Hopefully, it goes as easily for @emilazy. ## Source Releases The 10.12 SDK builds several frameworks and libraries from source. It also makes use of their headers. The new SDK pattern no longer tries to get headers from the source releases. They’re just too inconsistent with the headers in the SDK. Some headers are even older than the 10.12 because newer sources are not available or complete enough to use. However, this update actually builds and uses _more_ source releases than before. The following libraries are built. Of these, libresolv is new on aarch64-darwin, libsbuf is new for both (ported from FreeBSD due to the lack of a source release), libpcap is new for both, and copyfile and removefile are built from source for the first time. * ICU * libiconv * libresolv * libpcap * libsbuf * libutil * copyfile * removefile * xar Currently, libiconv and libresolv are propagated with the SDK on both platforms. For 25.05, libpcap and ICU may be made to be the default implementation on Darwin due to containing Apple-specific changes. It is also worth noting: all source releases correspond to the source released for macOS 15. When possible, it has been backported to build with the default SDKs. ### Source release version updates * AvailabilityVersions: 140.1 -> 143.3 * ICU: 66108 -> 74221 * IOKitTools: init at 125 * PowerManagement: 572.50.1 -> 1740.0.7 * adv_cmds: 163 -> 231 * basic_cmds: 55 -> 70 * bootstrap_cmds: 121 -> 136 * copyfile: 138 -> 213 * developer_cmds: 62 -> 79 * diskdev_cmds: 593 -> 735 * doc_cmds: init at 66 * file_cmds: 264.1.1 -> 448.0.3 * libiconv: 99 -> 107 * libpcap: init at 135 * libresolv: 64 -> 83 * libutil: 47.30.1 -> 72 * mail_cmds: init at 38.0.1 * misc_cmds: init at 44 * network_cmds: 606.40.2 -> 696 * patch_cmds: init at 62 * remote_cmds: init at 303.0.2 * removefile: 45 -> 75 * shell_cmds: 187 -> 319.0.1 * system_cmds: 970.0.4 -> 1012 * text_cmds: 99 -> 190.0.1 * top: 108 -> 139 * xattr: replace with xattr from file_cmds * xar: 498 -> 501 ## Transition Strategy Because this is a non-trivial change to how Darwin works, a transition is necessary. This update tries to avoid breaking existing packages, but there are breaking changes (see below). To minimize breakage, all existing SDK frameworks and libraries as well as dropped source releases (anything not above) have been replaced with stubs. These stubs contain a `README` advising the user to read the documentation on how to update to the new pattern. Currently, these stubs do nothing other than provide stubs. After a treewide has been done, they will moved to `darwin-aliases.nix`. It is expected that these aliases will remain in place until at least the 26.05 release. ### Breaking Changes * libiconv is no longer linked automatically. Since it is propagated by the SDK by default, including this hook would link libiconv to everything, which may not be desirable. Failing to link libiconv is a build system error. The upstream builds should be fixed to correct the mistake. * CoreFoundation is no longer linked automatically. Same as above. This is a build system issue. Hooks really shouldn’t be used to work around mistakes upstream, especially when the issues are not pervasive. * Anything that depended on the layout of the old frameworks will need to be updated to work with the new pattern: * Frameworks can be found at `SDKROOT/System/Library/Frameworks`; * The SDKROOT for a particular SDK package is available using the `sdkroot` attribute on the package (e.g., `apple-sdk.sdkroot`). * Darwin is now a “system” libc++ platform. LLVM will now always use the libc++ built in the stdenv unless you use `llvmPackages.libcxxStdenv`. This aligns Darwin with Linux where `llvmPackages.stdenv` uses the default libstdc++. * Darwin’s libc is determined dynamically. `darwin.libSystem` contains empty `lib` and `include` folders to avoid compiler and linker warnings that may be turned into errors. The libSystem stubs can be located at `$SDKROOT/usr/lib/libSystem.tbd`. * Packages may try to build additional functionality due to the presence of SDK frameworks. For example, Python no longer supports the `configd` option to disable the `_scproxy` module. The SDK always has the SystemConfiguration framework. * Vendored packages have been removed from the SDK. Between all the different versions, more than 50 packages are vendored. Those that are conventionally added as inputs are excluded from the SDK. Just add them as inputs like normal. Those that are part of the SDK are propagated. This includes libiconv and libresolv as well as the headers for ncurses (which are needed by Swift). Note that libutil is not propagated because its headers may cause problems with some packages (particularly Python). If you need libutil, add `darwin.libutil` as a build input. * Old source releases are no longer available, but their source for the SDK is accessible via the `apple-sdk.sourceRelease` function. This is mostly useful for copying private headers for use to build a derivation. See various source releases for examples. ## Closed Issues and PRs - closes #86299 - closes #101229 - closes #229210 - closes #242666 - closes #264091 - closes #269646 - closes #278348 - closes #273442 - closes #284629 - closes #308366 ## Things Still to be Done This PR is almost completely ready. If need be, it could be merged into staging to make the next staging-next cycle. The following. - [ ] Rework locale data. In a pinch, I could duplicate the current logic. Apple appears to be using the same source of data as FreeBSD, but the data from the FreeBSD doesn’t actually work with their tools. Honestly, I’m not sure setting the environment variable even works. - [ ] `netstat` doesn’t print active Internet connections. It prints other kinds but not those. It requires further investigation. - [ ] Bootstrap tools need updated. This could wait until after this is merged, but it will need to be done before this hits master to prevent the channel from being blocked. ## Related PRs These issues were found during testing. This [flake](https://gist.github.com/reckenrode/58933e82c687d8026fe19630c0ed699f) was used to help find issues. Some of these can target staging or even master without waiting. Others depend on this PR or even other PRs. (list will be updated as open PRs.) - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [x] x86_64-darwin - [x] aarch64-darwin - For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] `sandbox = relaxed` - [ ] `sandbox = true` - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [x] Tested basic functionality of all binary files (usually in `./result/bin/`) - [24.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) (or backporting [23.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) and [24.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2405.section.md) Release notes) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).  --- Add a :+1: [reaction] to [pull requests you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc