The Darwin SDKs have been updated

Announcements
1

A new Darwin SDK pattern has been merged into master.

Highlights of the new Pattern

SDKs Added and Source Release Updates

The following Darwin SDKs are now available in nixpkgs on master and should be available on unstable soon. They will be included in 24.11.

  • apple-sdk: 10.12.2 (x86_64-darwin) and 11.3 (aarch64-darwin)
  • apple-sdk_10_12: 10.12.2
  • apple-sdk_10_13: 10.13.2
  • apple-sdk_10_14: 10.14.6
  • apple-sdk_10_15: 10.15.6
  • apple-sdk_11: 11.3
  • apple-sdk_12: 12.3
  • apple-sdk_13: 13.3
  • apple-sdk_14: 14.4
  • apple-sdk_15: 15.0

The following source releases have also been updated. Going forward, source releases will track the latest source releases available.

  • AvailabilityVersions: 140.1 → 143.3
  • ICU: 66108 → 74221
  • IOKitTools: init at 125
  • PowerManagement: 572.50.1 → 1740.0.7
  • adv_cmds: 163 → 231
  • basic_cmds: 55 → 70
  • bootstrap_cmds: 121 → 136
  • copyfile: 138 → 213
  • developer_cmds: 62 → 79
  • diskdev_cmds: 593 → 735
  • doc_cmds: init at 66
  • file_cmds: 264.1.1 → 448.0.3
  • libffi: init at 35
  • libiconv: 99 → 107
  • libpcap: init at 135
  • libresolv: 64 → 83
  • libutil: 47.30.1 → 72
  • mail_cmds: init at 38.0.1
  • misc_cmds: init at 44
  • network_cmds: 606.40.2 → 696
  • patch_cmds: init at 62
  • remote_cmds: init at 303.0.2
  • removefile: 45 → 75
  • shell_cmds: 187 → 319.0.1
  • system_cmds: 970.0.4 → 1012
  • text_cmds: 99 → 190.0.1
  • top: 108 → 139
  • xattr: replaced with xattr from file_cmds
  • xar: 498 → 501

The following source releases have been removed. They are no longer used to build the SDK. Their corresponding headers and stubs are included in the SDK.

  • CF (darwin.swift-corelibs-foundation is still available)
  • CarbonHeaders
  • CommonCrypto
  • CoreSymbolication
  • IOKit
  • Libc
  • Libinfo
  • Libm
  • Libnotify
  • Librpcsvc
  • Libsystem
  • LibsystemCross
  • Security
  • architecture
  • configd (previously built a framework from source, but it is no longer buildable)
  • configdHeaders
  • darwin-stubs (already included in the SDK)
  • dtrace (the stubs are included in the SDK)
  • dyld (may return if dyld_info can be built and if ld-prime is ever made publicly available)
  • eap8021x
  • hfs
  • hfsHeaders
  • launchd
  • libclosure
  • libdispatch
  • libmalloc
  • libplatform
  • libpthread
  • mDNSResponder
  • objc4
  • ppp
  • xnu

About the New SDK Pattern

  • Instead of providing separate SDK frameworks and libraries, they are bundled together in an SDK that can be used with xcrun and the xcbuild package.
  • The SDK includes everything you need to build applications. It also propagates certain packages automatically (libiconv, libresolv, and libsbuf). It is no longer necessary to include these conditionally on Darwin.
  • The stdenv provides a default SDK (the unversioned apple-sdk package). A different SDK can be used to change the SDK in use by adding it to your buildInputs (see the stdenv documentation for more details).

xcbuild changes

  • xcbuild has also been updated to the last commit before the repo was archived.
  • It is no longer built with a wrapper but relies on the active SDK in the build environment to work.
  • You no longer have to override the sdkVersion in xcbuild to change the SDK. Just add the required SDK to your buildInputs.
  • The stdenv now provides xcrun. It is no longer necessary to patch out xcrun from build systems, though references to /usr/bin/xcrun should be replaced with ${lib.getExe' xcbuild "xcrun"}.
  • xcrun will find any package’s binaries in the build environment PATH.

Known Issues

Future Updates

This new pattern was developed to make updating both the SDKs and the source releases easier. The original PR added up through the 14.4 SDK. @emily added the 15.0 SDK in a separate PR. She indicated it took less than an hour even with the process being undocumented and having bugs. Future SDKs should also be just as easy to add. The 15.1 SDK is waiting on source release updates and will be added for 25.05.

Thanks and Gratitudes

Many thanks to @toonn, @emily, @paparodeo, and all the others who helped review, test, and provide feedback on this work. I’d also like to express my gratitude to everyone who worked in the past to make Darwin support possible and provided the foundation on which I was able to do this work.

Regarding the Old SDK Pattern

  • The old SDKs have been replaced by stubs containing a README advising you to read the stdenv documentation to update. Packages should eval, but the stubs do nothing. The default SDK in the stdenv should allow packages using the legacy SDK to build.
  • They were made inert to avoid accidentally mixing SDK versions, which has been a persistent problem in the past that made working with non-default Darwin SDK versions difficult to impossible depending on the language and build system.
  • darwin.apple_sdk_11_0.callPackage and overrideSDK have been updated to add the corresponding SDK to buildInputs for you. You should migrate to doing that directly because they are deprecated and will be removed eventually.

Transition

  • 25.05 is switching to the 11.3 SDK by default on both platforms. All older SDKs will be dropped, including darwin.apple_sdk_10_12.
  • Early in 25.05, the existing nixpkgs packages will be converted to use the new pattern. Once that has been done, the stub packages will be added to darwin-aliases.nix to prevent them from being used in nixpkgs.
  • For external users, these stubs will be updated print a warning about the removal. After this warning has been in place for at least two releases, the stubs and the old SDK packages will be removed (soonest: for 26.05).

Breaking Changes

  • Anything that depends on the internals of how the old SDK frameworks were laid out will need updating for the new SDK pattern. See the stdenv documentation for more details.
  • Darwin’s libc is determined dynamically based on the SDK. darwin.libSystem is a mostly empty package that provides empty include and lib folders to avoid warnings and errors about missing include and library search paths.
  • CoreFoundation and libiconv no longer provide hooks to automatically link them. Generally, this is an issue with the upstream build system that should (ideally) be fixed upstream. According to repology, there are over 100k packages in nixpkgs. Less than two dozen packages had to be fixed for this change. In exchange, libiconv can be (and is now) included in the SDK, allowing all (often Rust) packages currently adding libiconv conditionally on Darwin to drop it.
  • The version of xcrun in the xcbuild package is no longer a wrapper script. It is the binary from xcbuild, which looks up packages in the build environment’s PATH. The SDK’s toolchain includes only a minimal number of tools to
    allow Chromium-based packages to build (libtool from cctools, lipo, nm, otool, size, and strip). The following tools are no longer included in the toolchain: bison, flex, m4, unifdef, gperf, indent, ctags, mig and migcom (from bootstrap_cmds), mkdep (please open an issue if you need it; should be from developer_cmds, but it was removed upstream). If your package needs any of these commands, add them to nativeBuildInputs.
  • The SDK includes headers for CUPS (needed by some frameworks) and ncurses (needed by Swift’s modules). These headers may incorrectly cause a package to detect CUPS or ncurses. If you need to link one of these packages (or any of the ~50 packages vendored by the upstream SDK that are no longer included int he stubs), you should add as buildInputs.
  • Packages that do feature detection for Darwin frameworks will always detect them. If this is not wanted or causes problems with your package, you will have to disable the unwanted features manually.

Potential Future Changes

  • There are some improvements that could be made to xcbuild, but they would require forking it into a new repo.
  • cctools and ld64 will be updated to build on Linux to support Linux to Darwin cross.
  • darwin.libpcap and darwin.ICU may be made the default implementations on Darwin will be explored for 25.05. Both are built in a way that is compatible with top-level package expectations (including .pc files, etc) while also including extra Darwin functionality.

Updates:

62 Likes
2

This is huge news. Thank you so much for this!

3 Likes
3

Thanks so much! This is very cool.

Just a tip for comms: this post should probably start with the highlights and then describe the changes. As a user who is not a package developer, I didn’t make sense of it until I read about the impact on the longstanding issues like Firefox and Swift builds.

3 Likes
4

Amazing work, this is front page material!

Finally, Nixpkgs is the macOS software distribution I wanted to exist since I discovered Homebrew.

Thank you so much for making this happen!

11 Likes
5

I can’t emphasize enough what a big deal Randy’s work is here. 24.11 is going to be the best release of Nixpkgs for macOS ever, and 25.05 will be even better (11.3 bump and I’m hoping to do something about the ofborg situation). Hopefully we can grow the new Darwin team as well!

19 Likes
6

Thanks for the suggestion. I moved it to a new section at the top. I kind of buried the lede on Darwin to Darwin cross also being fixed by this update. :sweat_smile:

4 Likes
7

Thank you. Absolutely incredible work. If you don’t mind sharing ways that we can help you, I’d love to try to do so in the weeks and months ahead.

4 Likes
8

Off the top of my head, these are some of things coming that could always use help. If anyone is interested, I suggest dropping by the macOS channel on Matrix.

Swift is also in need of some love and attention, but that’s a pretty big project on its own. Not only does the derivation need reworked, it needs updated to Swift 5.10.1. After that, Swift 6 can be added (which requires a working Swift compiler to bootstrap).

5 Likes
9

So happy to see this has landed! A huge thank you to everyone involved!

2 Likes
10

This is awesome, great job and thank you maintainers!

Is there a way to identify these packages and help contribute to the cleanup?

2 Likes
11

Unfortunately, not easily. Two approaches come to mind:

  • Move the legacy SDKs to darwin-aliases.nix then fix every error that results when eval’ing with aliases disabled.

  • Grep for apple_sdk. That will reveal any packages in all-packages.nix using the inherit (darwin.apple_sdk.…) pattern as well as those using darwin.apple_sdk in their build inputs.

A good starting point for maintainers are any packages you maintain. I got the important ones I maintain, but there are some others needing cleanup. Note than this cleanup will cause rebuilds because you are changing inputs.

1 Like
12

Another possible set of cleanups are all the packages (particularly Rust) conditionally adding libiconv to their build inputs on Darwin.

2 Likes
13

As a macOS-only Nix user: thank you thank you thank you :infinity:. Such pivotal work.

4 Likes
14

A thousand thank you for doing this, feel free to claim 10000 USD from the macos fundrasier. This is why people have been donating and I truly believe you deserve it. Thus, a ten thousand thank you.

33 Likes
15

Seconding this. This kind of work is why we’ve been paying into that pool at work.

7 Likes
16

So guys, for a non darwin user until last couple of days, what do I need to do regarding this update? Did a nix flake update, nix-collect-garbage…

I’m particulary interested in this:

  • Wezterm now builds on x86_64-darwin.

As I’ve been trying to setup a new machine, have nix darwin flake setup. All other packages installed successfully except this wezterm.

17

I’m really flattered. I’ve been declining these offers out of concern for creating a conflict of interest at work (because of another engineer who left a few years ago due to one), but I was able to get clarification on our policies. I submitted the invoice. Thank you so very much. :heart:

17 Likes
18

It should show up on unstable once the channel advances. If you don’t want to wait, you can set your flake’s nixpkgs input to the master branch (with the usual caveat that sometimes master can be broken or possibly not have cached builds yet).

1 Like
19

There’s a few packages in poetry2nix which do this, if anyone wants to flex a python + rust + darwin + nix muscle.

1 Like
20

Thank you for your work.

For someone who is not MacOS user, but oftentimes needs to get Nix to work with MacOS (e.g. xcompilation to iOS etc.) what are the best resources to understand all the details involved, examples, etc.

2 Likes