Darwin Updates News

Announcements
1

This thread is a place to share updates to Darwin separate from Nix šŸ–¤ macOS Monthly to avoid creating confusion about which work is sponsored and which is not.

11 Likes
2

This is a repost from Nix šŸ–¤ macOS Monthly - #50 by reckenrode.

This PR updates system_cmds to 970.0.4 on both aarch64-darwin and x86_64-darwin. This version corresponds to the system_cmds in macOS 14, but they are built with the default SDKs and should work on older systems. Five binaries were lost in the update:

  • makekey - removed upstream
  • login - requires entitlements and non-public headers
  • sadc - removed upstream
  • sar - removed upstream
  • zprint - requires IOKitUser framework that isnā€™t packaged

Several more were added:

  • arch (aarch64-darwin only)
  • chpass
  • cpuctl
  • dynamic_pager
  • iosim
  • kpgo
  • ltop
  • memory_pressure
  • mslutil (aarch64-darwin only)
  • pagesize
  • passwd
  • proc_uuid_policy
  • purge
  • stackshot (aarch64-darwin only)
  • taskpolicy (aarch64-darwin only)
  • vm_purgeable_stat (aarch64-darwin only)
  • wait4path
  • wordexp-helper
  • zlog (aarch64-darwin only)

The aarch64-darwin only ones are due to the default SDK. If you override the SDK, they should build and work on x86_64-darwin. The following will also be available with the 12.0 SDK (once available):

  • lskq
  • lsmp
  • reboot - if/when IOKitUser framework is added
  • shutdown - if/when IOKitUser framework is added
2 Likes
3

Once the current staging-next cycle completes, Darwin will no longer require curl in the stdenv bootstrap build. It was only needed previously for fetchers, which can use the one bundled in the bootstrap tools tarball, and for hatch-vcs tests.

2 Likes
4

@a-n-n-a-l-e-e fixed the issue with linking libc++abi on Darwin. All of the hacks that were put in place during the LLVM 16 update and final staging-next cycle before the release of 23.11 have been removed.

5 Likes
5

The darwin attrset now supports aliases, which will allow packages to be moved and deprecated.

2 Likes
6

Darwin is now using the upstream libiconv. It is built with a compatible ABI, so it can be used with install_name_tool to replace references to the system libiconv with one from nixpkgs.

Update: This PR had to be reverted. It caused libgit2ā€™s tests to fail due to a dependency on a Mac-specific UTF-8 encoding for handling precomposed characters. I have a replacement in the works, but it needs to be reviewed and merged first before Iā€™ll post it here.

2 Likes
7

A rewrite of overrideSDK was merged into staging today. This version improves the robustness of overrideSDK to handle all inputs and (hopefully) play nicely with cross-compilation. It also sets the SDK root, which results in a binary being linked with the correct SDK version. On master, binaries will be linked with the SDK set to the deployment target due to how clang infers the SDK version.

Setting the SDK version correctly is important for two reasons:

  • macOS changes behavior or enables functionality depending on the linked SDK. For example, dark mode support requires a newer SDK even with an older back deployment target; and
  • It affects how the OS version is reported. I am working on adding patches for D3DMetal support in the nixpkgs version of Wine. D3DMetal requires macOS 14. The version check fails when linked with the current stdenv because SDK versions older than 11 return a compatibility version it doesnā€™t recognize (10.20 instead of 14.x).

It is planned to update rustPlatform to handle overriding better than it does currently. This is needed for applications like wezterm, neovide, and lapce to build currently with the newer SDK. After that, Iā€™ll be updating the documentation to reflect a preference for overrideSDK over darwin.apple_sdk_11_0.callPackage.

Note: overrideSDK is a stdenv adapter. Use it as follows. The attrset can be replaced with the desired SDK version for brevity. overrideSDK only supports Darwin, so you will need to make it conditional on stdenv.isDarwin if you need to support other platforms.

The version number will become a constraint in the future. It will match the newest version up to the next major version. "11.0" is the same as "^11.0" (though the latter syntax is not currently supported). When the 11.0 SDK is updated to the 11.3 SDK, "11.0" (or "^11.0" will match 11.3. Currently, you should use "11.0". The deployment target is passed through as-is.

stdenv = overrideSDK stdenv {
  darwinMinVersion = <deployment target>;
  darwinSdkVersion = <SDK version>;
};
# or (using the platformā€™s default deployment target)
stdenv = overrideSDK stdenv "11.0";
# or (to set just the deployment target)
stdenv = overrideSDK stdenv { darwinMinVersion = "10.7"; }
6 Likes
8

The default version of GCC on Darwin will be GCC 13 after the next staging-next cycle.

2 Likes
9

This one is more of an announcement than news. 24.11 will be updating the SDK on x86_64-darwin. This is anticipated to be done after the SDK refactor lands post-24.05. The specific version has not yet been determined. It will likely be either 10.13 or 10.14. Which version will be based on the needs of software in the bootstrap (e.g., LLVM doesnā€™t really support 10.12 anymore, and Darwin will be updating to LLVM 18 for 24.11).

7 Likes
10

I am working on updates to cctools, ld64, and libtapi. While itā€™s possible they may land for 24.05, itā€™s very likely they wonā€™t make the freeze in time. I wanted to post something because this is a potentially bit update, and I havenā€™t posted news in a while.

Packaging Changes

  • cctools, ld64, and libtapi are now built from Appleā€™s OSS distributions repo instead of the cctools-port and apple-libtapi repos.
  • These versions (cctools 1010.6, ld64 951.9, libtapi 1500.0.12.3) correspond to the versions shipped with Xcode 15. cctools and ld64 are current as of Xcode 15.3. Packages should no longer have to patch around missing flags or feature support.
  • Restoring Linux support is a WIP. I have made some changes to use non-private and non-portable APIs, but I am currently focused on getting Darwin updates first.
  • The packages have been moved to the by-name hierarchy. Aliases are provided to the old names, but there are some breaking changes (see below).
  • cctools is now only cctools. This is technically a breaking change (see below). ld64 has been moved to its own, separate package.
  • darwin.binutils is now the package that defines the mix of cctools and LLVM tools. It has been update with additional tools. See below for the breakdown of what it contains.
  • The build system has been replaced with Meson. I did this because dealing with xcbuild stinks, and these packages eventually need to support Linux for cross-compilation.
  • corecrypto and CommonCrypto hashing APIs are reimplemented using OpenSSL on both Darwin (and eventually) Linux. cctools-port only does this on non-Darwin platforms.
  • libtapi has been split into separate out and lib outputs. Most packages only need libtapi.dylib, which is ~500 KiB compared to ~30 MiB for the tapi binary.
  • libtapi is linked against zlib. I donā€™t know what it uses zlib to do, but the upstream libtapi is also linked against zlib, and I want to match what upstream is doing.

Note: ld-prime is will not be packaged because it is not currently included in the source releases. While I think itā€™s unlikely, ld-prime will be added separately as a new package should the source be released.

New Features

Breaking Changes

  • ld64 and cctools use GCD (libdispatch) for parallelism. This should improve performance, but it technically ups the required macOS version to 10.14. The APIs it uses are present under different names in 10.12, so I have attempted to patch around them. It builds with the 10.12 SDK, but it will need testing to confirm the tools actually work. This was the driver of updating the default SDK on x86_64-darwin for 24.11.
  • Apple removed support for nested static archives from ld64. This happened with the release of Xcode 14, so it has been out in the ecosystem for a few years. Hopefully packages have adapted by now. For example, Kyua adds libutils.a to libcli.a using libcli_a_LIBADD in automake. This will fail to link with ld64 951.9.
  • I have chosen not to build GNU as in cctools. It is deprecated upstream. The default in nixpkgs is to use the clang integrated assembler (matching upstream). I am aware that GNAT is documented as requiring GNU as for its bootstrap. I plan to investigate and make changes accordingly to make sure GNAT still builds.
    • Update: This has changed. GNU as is built, but it is now found in the gas output on cctools. darwin.binutilsDualAs has been redefined as a buildEnv of darwin.binutils and cctools.gas, so packages that need both should continue to build. Those that need gas alone can add cctools.gas as an input.
  • cctools only provides binaries from cctools. ld64 is a separate package now.
  • darwin.cctools-apple, darwin.cctools-port, and darwin.cctools-llvm are replaced by just cctools. cctools-llvm will be an alias to darwin.bintools.
  • The old names canā€™t be used in nixpkgs because aliases are disabled when ofborg eval is run. The PR updating the tools will include changes for all packages using the old names. (Updates for LTO will be done in a separate PR or PRs.)
  • Use of postLinkSignHook has been removed from bintools-wrapper. Because it removes linker-created signatures, it breaks the automatic updating of signatures by strip and install_name_tool. Unless youā€™re manually invoking codesign, you shouldnā€™t be affected. install_name_tool will do the right thing by default.
  • strip and install_name_tool are no longer wrapped. If you are modifying linker-signed binaries, which should be the typical case, they will update the signatures automatically. If you are manually invoking codesign, you will need to update the signatures manually after running strip or install_name_tool.

stdenv updates

  • Stage 2 has been collapsed from two parts into one stage. The source-based CF is not coming back. It adds a lot of complexity to the bootstrap.
  • As noted in breaking changes, the Darwin stdenv no longer provides bintools using postLinkSignHook. Once the bootstrap tools are updated with these releases, it may be possible to drop sigtool completely from the bootstrap. Until that happens, early stages manually implement postLinkSignHook because the bootstrap tools ld64 does not appear capable of signing binaries on its own.
    As part of this update, I have to update the stdenv to use the new paths.
  • The overall stdenv size is down by ~20 MiB due to dropping tapi from the closure. It also drops signtool and signingUtils, but those binaries are small.
  • curl and GNU binutils throw when used during the bootstrap. curl should never be used during the bootstrap (to allow it to be updated without triggering a whole rebuild of Darwin). GNU binutils generally shouldnā€™t be used on Darwin (e.g., Rust 1.77 will clash harder with GNU binutils on Darwin Nix Ā· Issue #299606 Ā· NixOS/nixpkgs Ā· GitHub), so the bootstrap shouldnā€™t need to use it.

darwin.bintools changes

  • Variants of darwin.bintools will likely be removed. There is only darwin.bintools and darwin.binutils (for wrapped versions similar to top-level binutils). If GNAT still requires it, something will be done (tbd) to ensure it still has access to GNU as.
  • Most packages are from LLVM. c++filter is no longer linked from GNU binutils. It is linked from LLVM. A handful of packages are linked from cctools because LLVM does not provided drop-in replacements for them. The lists of tools are exported as attributes on darwin.bintools (llvm_cmds and cctools_cmds respectively). All binaries are symlinked to their traditional names (e.g., llvm-strip is symlinked to strip).
    • LLVM tools: addr2line, ar, c++filt, dsymutil, lipo, install-name-tool, nm, objcopy, objdump, otool, size, strings, strip
    • cctools tools: codesign_allocate, gprof, ranlib (and ld from ld64)
  • Darwin is still isCCTools and not isLLVM because it does not use lld by default. Maybe that could change with lld 18, but Iā€™m doubtful.

Iā€™ll announce here once I have a branch available for testing. Iā€™ll also see if I can get a Hydra job for it because I want to check for any build failures.

18 Likes
11

As a follow up to Darwin Updates News - #6 by reckenrode, libiconv has been updated for Darwin to libiconv-99, which matches the version of libiconv shipped in macOS 14.4. Updating Appleā€™s libiconv was necessary because of the need for the special, Mac-specific UTF-8 encoding. This libiconv implementation is based on the one from FreeBSD.

In addition to the libiconv update, three packages have been introduced that may be of interest to FreeBSD users: ATF, Kyua, and Lutok. The versions of these packages in nixpkgs should correspond to the current versions in the FreeBSD ports tree.

1 Like
12

I have opened the draft PR for the cctools and ld64 updates. I will be rebasing it weekly on staging and pushing once the Darwin blockers channel and my configs build, which takes about a day and a half on an M3 Max.

I plan to take a break from this and look at rebasing the Darwin cross PR on this branch then get back to cherry-picking some of the commits into separate PRs. The libunwind change will probably require an upstream PR to add the missing pkg-config file.

7 Likes
13

Darwin will be updating to clang 18 for 24.11, so I have opened the issue to track updates. Depending on the timing, some of the updates may make it into 24.05 since they donā€™t cause breaking changes. The plan is to open the PR to update the version once I have all of the Darwin blockers building and 24.05 has released.

3 Likes
14

The 12.3 SDK has been added to master.

  • It is usable via overrideSDK. The apple_sdk.callPackage pattern is not supported.
  • The overrideSDK adapter is only for Darwin stdenvs. If a package supports Linux, the override should be done conditionally.
  • SDK frameworks should used from the unversioned apple_sdk.frameworks. Versioned frameworks (e.g., apple_sdk_11_0.frameworks.CoreAudio) are not overridden by overrideSDK.

For an example of a package using the 12.3 SDK, see MoltenVK, which has been updated to use it.

Requesting an SDK version:

stdenv = overrideSDK stdenv "12.3";

Requesting an SDK version with a deployment target.

stdenv = overrideSDK stdenv {
  darwinMinVersion = "10.15";
  darwinSdkVersion = "12.3";
};

Note: This is an interim addition while the SDK refactor is being done to allow packages requiring a newer SDK to be built.

11 Likes
15

The cctools and ld64 update has been merged into staging. I will be following up with PRs to enable LTO on packages where it is disabled and to implement the treewide changes needed to move cctools to darwin-aliases.nix.

5 Likes
16

Two follow-ups to the cctools/ld64 update have been merged into staging.

The second PR also enables LTO for Python, which results in a ~10% performance improvement on py-bench.

9 Likes