Darwin Updates News

[reckenrode]

This thread is a place to share updates to Darwin separate from Nix 🖤 macOS Monthly to avoid creating confusion about which work is sponsored and which is not.

[reckenrode]

This is a repost from Nix 🖤 macOS Monthly - #50 by reckenrode.

darwin.system_cmds: 735.50.6 -> 970.0.4

This PR updates system_cmds to 970.0.4 on both aarch64-darwin and x86_64-darwin. This version corresponds to the system_cmds in macOS 14, but they are built with the default SDKs and should work on older systems. Five binaries were lost in the update:

• makekey - removed upstream
• login - requires entitlements and non-public headers
• sadc - removed upstream
• sar - removed upstream
• zprint - requires IOKitUser framework that isn’t packaged

Several more were added:

• arch (aarch64-darwin only)
• chpass
• cpuctl
• dynamic_pager
• iosim
• kpgo
• ltop
• memory_pressure
• mslutil (aarch64-darwin only)
• pagesize
• passwd
• proc_uuid_policy
• purge
• stackshot (aarch64-darwin only)
• taskpolicy (aarch64-darwin only)
• vm_purgeable_stat (aarch64-darwin only)
• wait4path
• wordexp-helper
• zlog (aarch64-darwin only)

The aarch64-darwin only ones are due to the default SDK. If you override the SDK, they should build and work on x86_64-darwin. The following will also be available with the 12.0 SDK (once available):

• lskq
• lsmp
• reboot - if/when IOKitUser framework is added
• shutdown - if/when IOKitUser framework is added

[reckenrode]

Once the current staging-next cycle completes, Darwin will no longer require curl in the stdenv bootstrap build. It was only needed previously for fetchers, which can use the one bundled in the bootstrap tools tarball, and for hatch-vcs tests.

darwin.stdenv: drop curl from the bootstrap

[reckenrode]

@a-n-n-a-l-e-e fixed the issue with linking libc++abi on Darwin. All of the hacks that were put in place during the LLVM 16 update and final staging-next cycle before the release of 23.11 have been removed.

llvmPackages_{12,13,14,15,16,17,git}.{libcxx,libcxxabi}: merge libcxxabi into libcxx

[reckenrode]

The darwin attrset now supports aliases, which will allow packages to be moved and deprecated.

darwin: add darwin-aliases.nix

[reckenrode]

Darwin is now using the upstream libiconv. It is built with a compatible ABI, so it can be used with install_name_tool to replace references to the system libiconv with one from nixpkgs.

Update: This PR had to be reverted. It caused libgit2’s tests to fail due to a dependency on a Mac-specific UTF-8 encoding for handling precomposed characters. I have a replacement in the works, but it needs to be reviewed and merged first before I’ll post it here.

darwin.libiconv: switch to libiconvReal and add to darwin-aliases

[reckenrode]

A rewrite of overrideSDK was merged into staging today. This version improves the robustness of overrideSDK to handle all inputs and (hopefully) play nicely with cross-compilation. It also sets the SDK root, which results in a binary being linked with the correct SDK version. On master, binaries will be linked with the SDK set to the deployment target due to how clang infers the SDK version.

Setting the SDK version correctly is important for two reasons:

• macOS changes behavior or enables functionality depending on the linked SDK. For example, dark mode support requires a newer SDK even with an older back deployment target; and
• It affects how the OS version is reported. I am working on adding patches for D3DMetal support in the nixpkgs version of Wine. D3DMetal requires macOS 14. The version check fails when linked with the current stdenv because SDK versions older than 11 return a compatibility version it doesn’t recognize (10.20 instead of 14.x).

It is planned to update rustPlatform to handle overriding better than it does currently. This is needed for applications like wezterm, neovide, and lapce to build currently with the newer SDK. After that, I’ll be updating the documentation to reflect a preference for overrideSDK over darwin.apple_sdk_11_0.callPackage.

Note: overrideSDK is a stdenv adapter. Use it as follows. The attrset can be replaced with the desired SDK version for brevity. overrideSDK only supports Darwin, so you will need to make it conditional on stdenv.isDarwin if you need to support other platforms.

The version number will become a constraint in the future. It will match the newest version up to the next major version. "11.0" is the same as "^11.0" (though the latter syntax is not currently supported). When the 11.0 SDK is updated to the 11.3 SDK, "11.0" (or "^11.0" will match 11.3. Currently, you should use "11.0". The deployment target is passed through as-is.

stdenv = overrideSDK stdenv {
  darwinMinVersion = <deployment target>;
  darwinSdkVersion = <SDK version>;
};
# or (using the platform’s default deployment target)
stdenv = overrideSDK stdenv "11.0";
# or (to set just the deployment target)
stdenv = overrideSDK stdenv { darwinMinVersion = "10.7"; }

darwin.stdenv: improvements and overrideSDK rewrite

[reckenrode]

The default version of GCC on Darwin will be GCC 13 after the next staging-next cycle.

default-gcc-version: 12 -> 13 if stdenv.isDarwin

[reckenrode]

This one is more of an announcement than news. 24.11 will be updating the SDK on x86_64-darwin. This is anticipated to be done after the SDK refactor lands post-24.05. The specific version has not yet been determined. It will likely be either 10.13 or 10.14. Which version will be based on the needs of software in the bootstrap (e.g., LLVM doesn’t really support 10.12 anymore, and Darwin will be updating to LLVM 18 for 24.11).

[reckenrode]

I am working on updates to cctools, ld64, and libtapi. While it’s possible they may land for 24.05, it’s very likely they won’t make the freeze in time. I wanted to post something because this is a potentially bit update, and I haven’t posted news in a while.

Packaging Changes

• cctools, ld64, and libtapi are now built from Apple’s OSS distributions repo instead of the cctools-port and apple-libtapi repos.
• These versions (cctools 1010.6, ld64 951.9, libtapi 1500.0.12.3) correspond to the versions shipped with Xcode 15. cctools and ld64 are current as of Xcode 15.3. Packages should no longer have to patch around missing flags or feature support.
• Restoring Linux support is a WIP. I have made some changes to use non-private and non-portable APIs, but I am currently focused on getting Darwin updates first.
• The packages have been moved to the by-name hierarchy. Aliases are provided to the old names, but there are some breaking changes (see below).
• cctools is now only cctools. This is technically a breaking change (see below). ld64 has been moved to its own, separate package.
• darwin.binutils is now the package that defines the mix of cctools and LLVM tools. It has been update with additional tools. See below for the breakdown of what it contains.
• The build system has been replaced with Meson. I did this because dealing with xcbuild stinks, and these packages eventually need to support Linux for cross-compilation.
• corecrypto and CommonCrypto hashing APIs are reimplemented using OpenSSL on both Darwin (and eventually) Linux. cctools-port only does this on non-Darwin platforms.
• libtapi has been split into separate out and lib outputs. Most packages only need libtapi.dylib, which is ~500 KiB compared to ~30 MiB for the tapi binary.
• libtapi is linked against zlib. I don’t know what it uses zlib to do, but the upstream libtapi is also linked against zlib, and I want to match what upstream is doing.

Note: ld-prime is will not be packaged because it is not currently included in the source releases. While I think it’s unlikely, ld-prime will be added separately as a new package should the source be released.

New Features

• LTO support. With this update, LTO works. I had to patch ld64 to look for libLTO.dylib in the store and clang to pass the correct path (see llvmPackages_{9,12,13,14,15,16,17,18,git}.clang: fix libLTO.dylib path by reckenrode · Pull Request #302481 · NixOS/nixpkgs · GitHub). I bootstrap ld64 and link it with itself with LTO enabled to make sure that LTO actually works. Fixes Clang link time optimization not working on OSX · Issue #19098 · NixOS/nixpkgs · GitHub.
• Code directory support. cctools is built with libcodedirectory support, which allows strip and install_name_tool to identify linker-created signatures and update them automatically. This removes the need for postLinkSignHook and actually makes it harmful to continue using it by default (because it prevents automatic signature updates by replacing them with a non-linker-created signature).

Breaking Changes

• ld64 and cctools use GCD (libdispatch) for parallelism. This should improve performance, but it technically ups the required macOS version to 10.14. The APIs it uses are present under different names in 10.12, so I have attempted to patch around them. It builds with the 10.12 SDK, but it will need testing to confirm the tools actually work. This was the driver of updating the default SDK on x86_64-darwin for 24.11.
• Apple removed support for nested static archives from ld64. This happened with the release of Xcode 14, so it has been out in the ecosystem for a few years. Hopefully packages have adapted by now. For example, Kyua adds libutils.a to libcli.a using libcli_a_LIBADD in automake. This will fail to link with ld64 951.9.
• I have chosen not to build GNU as in cctools. It is deprecated upstream. The default in nixpkgs is to use the clang integrated assembler (matching upstream). I am aware that GNAT is documented as requiring GNU as for its bootstrap. I plan to investigate and make changes accordingly to make sure GNAT still builds.
  • Update: This has changed. GNU as is built, but it is now found in the gas output on cctools. darwin.binutilsDualAs has been redefined as a buildEnv of darwin.binutils and cctools.gas, so packages that need both should continue to build. Those that need gas alone can add cctools.gas as an input.
• cctools only provides binaries from cctools. ld64 is a separate package now.
• darwin.cctools-apple, darwin.cctools-port, and darwin.cctools-llvm are replaced by just cctools. cctools-llvm will be an alias to darwin.bintools.
• The old names can’t be used in nixpkgs because aliases are disabled when ofborg eval is run. The PR updating the tools will include changes for all packages using the old names. (Updates for LTO will be done in a separate PR or PRs.)
• Use of postLinkSignHook has been removed from bintools-wrapper. Because it removes linker-created signatures, it breaks the automatic updating of signatures by strip and install_name_tool. Unless you’re manually invoking codesign, you shouldn’t be affected. install_name_tool will do the right thing by default.
• strip and install_name_tool are no longer wrapped. If you are modifying linker-signed binaries, which should be the typical case, they will update the signatures automatically. If you are manually invoking codesign, you will need to update the signatures manually after running strip or install_name_tool.

stdenv updates

• Stage 2 has been collapsed from two parts into one stage. The source-based CF is not coming back. It adds a lot of complexity to the bootstrap.
• As noted in breaking changes, the Darwin stdenv no longer provides bintools using postLinkSignHook. Once the bootstrap tools are updated with these releases, it may be possible to drop sigtool completely from the bootstrap. Until that happens, early stages manually implement postLinkSignHook because the bootstrap tools ld64 does not appear capable of signing binaries on its own.
  As part of this update, I have to update the stdenv to use the new paths.
• The overall stdenv size is down by ~20 MiB due to dropping tapi from the closure. It also drops signtool and signingUtils, but those binaries are small.
• curl and GNU binutils throw when used during the bootstrap. curl should never be used during the bootstrap (to allow it to be updated without triggering a whole rebuild of Darwin). GNU binutils generally shouldn’t be used on Darwin (e.g., Rust 1.77 will clash harder with GNU binutils on Darwin Nix · Issue #299606 · NixOS/nixpkgs · GitHub), so the bootstrap shouldn’t need to use it.

darwin.bintools changes

• Variants of darwin.bintools will likely be removed. There is only darwin.bintools and darwin.binutils (for wrapped versions similar to top-level binutils). If GNAT still requires it, something will be done (tbd) to ensure it still has access to GNU as.
• Most packages are from LLVM. c++filter is no longer linked from GNU binutils. It is linked from LLVM. A handful of packages are linked from cctools because LLVM does not provided drop-in replacements for them. The lists of tools are exported as attributes on darwin.bintools (llvm_cmds and cctools_cmds respectively). All binaries are symlinked to their traditional names (e.g., llvm-strip is symlinked to strip).
  • LLVM tools: addr2line, ar, c++filt, dsymutil, lipo, install-name-tool, nm, objcopy, objdump, otool, size, strings, strip
  • cctools tools: codesign_allocate, gprof, ranlib (and ld from ld64)
• Darwin is still isCCTools and not isLLVM because it does not use lld by default. Maybe that could change with lld 18, but I’m doubtful.

I’ll announce here once I have a branch available for testing. I’ll also see if I can get a Hydra job for it because I want to check for any build failures.

[reckenrode]

As a follow up to Darwin Updates News - #6 by reckenrode, libiconv has been updated for Darwin to libiconv-99, which matches the version of libiconv shipped in macOS 14.4. Updating Apple’s libiconv was necessary because of the need for the special, Mac-specific UTF-8 encoding. This libiconv implementation is based on the one from FreeBSD.

In addition to the libiconv update, three packages have been introduced that may be of interest to FreeBSD users: ATF, Kyua, and Lutok. The versions of these packages in nixpkgs should correspond to the current versions in the FreeBSD ports tree.

darwin.libiconv: 50 -> 99 and move to pkgs/by-name

[reckenrode]

I have opened the draft PR for the cctools and ld64 updates. I will be rebasing it weekly on staging and pushing once the Darwin blockers channel and my configs build, which takes about a day and a half on an M3 Max.

I plan to take a break from this and look at rebasing the Darwin cross PR on this branch then get back to cherry-picking some of the commits into separate PRs. The libunwind change will probably require an upstream PR to add the missing pkg-config file.

cctools: 973.0.1 -> 1010.6