Darwin Updates News

Announcements
10

I am working on updates to cctools, ld64, and libtapi. While it鈥檚 possible they may land for 24.05, it鈥檚 very likely they won鈥檛 make the freeze in time. I wanted to post something because this is a potentially bit update, and I haven鈥檛 posted news in a while.

Packaging Changes

  • cctools, ld64, and libtapi are now built from Apple鈥檚 OSS distributions repo instead of the cctools-port and apple-libtapi repos.
  • These versions (cctools 1010.6, ld64 951.9, libtapi 1500.0.12.3) correspond to the versions shipped with Xcode 15. cctools and ld64 are current as of Xcode 15.3. Packages should no longer have to patch around missing flags or feature support.
  • Restoring Linux support is a WIP. I have made some changes to use non-private and non-portable APIs, but I am currently focused on getting Darwin updates first.
  • The packages have been moved to the by-name hierarchy. Aliases are provided to the old names, but there are some breaking changes (see below).
  • cctools is now only cctools. This is technically a breaking change (see below). ld64 has been moved to its own, separate package.
  • darwin.binutils is now the package that defines the mix of cctools and LLVM tools. It has been update with additional tools. See below for the breakdown of what it contains.
  • The build system has been replaced with Meson. I did this because dealing with xcbuild stinks, and these packages eventually need to support Linux for cross-compilation.
  • corecrypto and CommonCrypto hashing APIs are reimplemented using OpenSSL on both Darwin (and eventually) Linux. cctools-port only does this on non-Darwin platforms.
  • libtapi has been split into separate out and lib outputs. Most packages only need libtapi.dylib, which is ~500 KiB compared to ~30 MiB for the tapi binary.
  • libtapi is linked against zlib. I don鈥檛 know what it uses zlib to do, but the upstream libtapi is also linked against zlib, and I want to match what upstream is doing.

Note: ld-prime is will not be packaged because it is not currently included in the source releases. While I think it鈥檚 unlikely, ld-prime will be added separately as a new package should the source be released.

New Features

Breaking Changes

  • ld64 and cctools use GCD (libdispatch) for parallelism. This should improve performance, but it technically ups the required macOS version to 10.14. The APIs it uses are present under different names in 10.12, so I have attempted to patch around them. It builds with the 10.12 SDK, but it will need testing to confirm the tools actually work. This was the driver of updating the default SDK on x86_64-darwin for 24.11.
  • Apple removed support for nested static archives from ld64. This happened with the release of Xcode 14, so it has been out in the ecosystem for a few years. Hopefully packages have adapted by now. For example, Kyua adds libutils.a to libcli.a using libcli_a_LIBADD in automake. This will fail to link with ld64 951.9.
  • I have chosen not to build GNU as in cctools. It is deprecated upstream. The default in nixpkgs is to use the clang integrated assembler (matching upstream). I am aware that GNAT is documented as requiring GNU as for its bootstrap. I plan to investigate and make changes accordingly to make sure GNAT still builds.
    • Update: This has changed. GNU as is built, but it is now found in the gas output on cctools. darwin.binutilsDualAs has been redefined as a buildEnv of darwin.binutils and cctools.gas, so packages that need both should continue to build. Those that need gas alone can add cctools.gas as an input.
  • cctools only provides binaries from cctools. ld64 is a separate package now.
  • darwin.cctools-apple, darwin.cctools-port, and darwin.cctools-llvm are replaced by just cctools. cctools-llvm will be an alias to darwin.bintools.
  • The old names can鈥檛 be used in nixpkgs because aliases are disabled when ofborg eval is run. The PR updating the tools will include changes for all packages using the old names. (Updates for LTO will be done in a separate PR or PRs.)
  • Use of postLinkSignHook has been removed from bintools-wrapper. Because it removes linker-created signatures, it breaks the automatic updating of signatures by strip and install_name_tool. Unless you鈥檙e manually invoking codesign, you shouldn鈥檛 be affected. install_name_tool will do the right thing by default.
  • strip and install_name_tool are no longer wrapped. If you are modifying linker-signed binaries, which should be the typical case, they will update the signatures automatically. If you are manually invoking codesign, you will need to update the signatures manually after running strip or install_name_tool.

stdenv updates

  • Stage 2 has been collapsed from two parts into one stage. The source-based CF is not coming back. It adds a lot of complexity to the bootstrap.
  • As noted in breaking changes, the Darwin stdenv no longer provides bintools using postLinkSignHook. Once the bootstrap tools are updated with these releases, it may be possible to drop sigtool completely from the bootstrap. Until that happens, early stages manually implement postLinkSignHook because the bootstrap tools ld64 does not appear capable of signing binaries on its own.
    As part of this update, I have to update the stdenv to use the new paths.
  • The overall stdenv size is down by ~20 MiB due to dropping tapi from the closure. It also drops signtool and signingUtils, but those binaries are small.
  • curl and GNU binutils throw when used during the bootstrap. curl should never be used during the bootstrap (to allow it to be updated without triggering a whole rebuild of Darwin). GNU binutils generally shouldn鈥檛 be used on Darwin (e.g., Rust 1.77 will clash harder with GNU binutils on Darwin Nix 路 Issue #299606 路 NixOS/nixpkgs 路 GitHub), so the bootstrap shouldn鈥檛 need to use it.

darwin.bintools changes

  • Variants of darwin.bintools will likely be removed. There is only darwin.bintools and darwin.binutils (for wrapped versions similar to top-level binutils). If GNAT still requires it, something will be done (tbd) to ensure it still has access to GNU as.
  • Most packages are from LLVM. c++filter is no longer linked from GNU binutils. It is linked from LLVM. A handful of packages are linked from cctools because LLVM does not provided drop-in replacements for them. The lists of tools are exported as attributes on darwin.bintools (llvm_cmds and cctools_cmds respectively). All binaries are symlinked to their traditional names (e.g., llvm-strip is symlinked to strip).
    • LLVM tools: addr2line, ar, c++filt, dsymutil, lipo, install-name-tool, nm, objcopy, objdump, otool, size, strings, strip
    • cctools tools: codesign_allocate, gprof, ranlib (and ld from ld64)
  • Darwin is still isCCTools and not isLLVM because it does not use lld by default. Maybe that could change with lld 18, but I鈥檓 doubtful.

I鈥檒l announce here once I have a branch available for testing. I鈥檒l also see if I can get a Hydra job for it because I want to check for any build failures.

18 Likes