Nix 🖤 macOS Monthly

As far as I can tell, the command-line tools (not sure about Xcode) make an exception only for the open-source components (and I have no idea how that works when the open-source components are incomplete or unavailable). What about the other stuff though? How does that work for e.g., Metal or CoreImage?

April 2022

Something that stood out with the _scproxy issue was the involvement of a bootstrap stage Python in the bootstrapped-pip build. This raised my suspicion that something was wrong with the Python build. Turns out there were errors preventing _scproxy.c from being compiled but these errors were silently swallowed and did not result in a failing build.

The reason for the errors was an issue with the Availability macros. Apple stopped updating them in the open source releases, probably because of the combinatorial explosion of macros with each new version of macOS. They’ve since switched to newer macros, which are backed by the availability attribute in modern Clang and GCC.

My fix is redefining the old macros using the new-style availability attribute. This finally fixed the Python build and completed the substitution of configd by SystemConfiguration from the SDK. Now I can continue bumping Apple’s open source releases.

16 Likes

May 2022

With configd out of the way I was finally able to focus on bumping the actual Apple open source releases we rely on to construct our own source-based SDK. This went swimmingly. I’ve updated about half of the source releases and tracked down some of the headers we were missing, so our SDK will be more complete. So far I haven’t encountered any major, new roadblocks—knock on wood.

19 Likes

@toonn: I see these errors popping up again, after switching default python from 3.9 to 3.10 (currently on the staging-next branch). Very many packages seem to be transitively broken by that. Perhaps you could have a look?

1 Like

Has anyone dared to try upgrading to the Ventura developer beta yet? We’ve gotten lucky with the last 1 or 2 upgrades and things have mostly worked out of the box, but prior to that it’s often been a very painful process.

1 Like

We’ve been talking about this a bit in #macos:nixos.org. I’m looking into this today but there’s only so much I can do because I don’t have access to aarch64-darwin hardware.

What I can already say is that I don’t think the errors are related to the ones I observed. As aarch64-darwin is using SDK 11.0.1 it shouldn’t have issues with the Availability macros.

2 Likes

Right now, x86_64-darwin seems to be broken with the same error message.

1 Like

would you like access? happy to provide

Thanks for the offer, @pjjw, but the Python issue was tracked down to intentionally using python3Minimal for cacert to workaround an infinite recursion issue with mailcap. This has since been rolled back and I believe it was fixed by using fetchurl instead of fetchzip for mailcap, but this hasn’t been confirmed.

I’ll let you know if the need arises again : )

1 Like

June 2022

Continuing to bump source releases. hfs is the only one so far that I haven’t been able to update, newer versions miss headers present in the SDK. I managed to get more of shell_cmds building. Only sudo is still missing but in later versions it may require entitlements so we probably wouldn’t be able to keep it building anyway. I also made some changes to our header checks to include checking for symlinked headers, which are common in the SDK.

The Libc bump is more involved because it used to be an amalgam of three versions and I’m dropping all the old ones, which requires digging through the sources for missing headers. So far it seems viable to drop the older versions.

13 Likes

July 2022

I’m down to only a handful of source releases that still need bumping. Libc still requires one older version because Apple stopped shipping certain headers. The pkill and pgrep commands of adv_cmds still do not build, but colldef and mklocale do. One of the bigger challenges was network_cmds, it depends on OpenSSL. However, Apple’s OpenSSL for macOS 10.13 was at version 0.9.8. The network_cmds in Nixpkgs is currently built with OpenSSL 1.0.2 but the new version wasn’t compatible anymore. Rather than go back to Apple’s older version, I patched network_cmds for compatibility with OpenSSL 1.1.0. Eyes on this would be appreciated.

In other news, @reckenrode :heart: made it possible to use the 11.0 SDK on x86_64-darwin! This was prompted by their work on MoltenVK and additionally motivated by Go 1.18 requiring a newer SDK, not to mention various other projects like Python and Qt. For now the intent is for this to be used sparingly, only for cases where it’s unavoidable.

19 Likes

August 2022

I’ve bumped all of the Apple source releases, configd being the main missing piece. For now we will stick with SystemConfiguration from the Apple SDK as a substitute. This time around, system_cmds was the most troublesome project. I ended up having to drop gcore, lskq, lsmp and zprint but managed to fix the builds for dirhelper, dmesg, dynamic_pager, ltop, pagesize, proc_uuid_policy, taskpolicyandvm_purgeable_stat`, it remains to be seen whether those are requirements for any packages.

The next step is to do a full rebuild on Hydra to check how many failures are introduced by these changes. And then things need to go through review. I’d especially appreciate more eyes on the changes to network_cmds related to OpenSSL. @veprbl already pointed out an initialization issue but I’m sure this can use more scrutiny.

One current technical hurdle is Hydra is not currently building bootstrap-tools, but this is being worked on in PR #183072.

10 Likes

September 2022

Latest evaluation still has many failing jobs. I included one too many changes from darwin-stubs and those will only start working once the corresponding Nixpkgs PR is merged. The deadline for changes to release critical packages for 22.11 is coming up in less than two weeks. This doesn’t leave enough time for thorough reviewing. So a bit of a false start but on the bright side this should be the last release window missed.

Thanks to @a-m-joseph :heart: for fixing the bootstrap-tools SNAFU in PR #183072, this change is currently making its way through staging. The bootstrap-tools are failing on aarch64-darwin, which will block nixpkgs-unstable so there is more work to be done.

13 Likes

October 2022

The availability attribute I started using here is not compatible with GCC. This didn’t really come up before because Darwin stdenv uses Clang. However, many packages in Nixpkgs do use GFortran, which is really GCC so it did come up in the Hydra evaluation. The way to support availability macros for GCC is simply to go with Apple’s older approach of having macros for all the various combinations of versions.

I want to highlight @stephank’s work bringing Swift to Darwin. It’s a big change so more eyes to help along the review would be welcome.

9 Likes

November 2022

I finished implementing the old-style macros (comparison to trunk) and GCC and GFortran (except for 6 and older) now build. There are still many failing builds, some of these seem related to the builds running under Rosetta2 but there are other problems. RustC is failing to build (a bootstrap stage compiler is exiting unsuccessfully), Python’s requests is failing to build due to CFFI (Could this be related to #187636, since these builds are running under Rosetta2 on aarch64-darwin?), Libepoxy is another common dependency failing due to a segmentation violation. So lots of issues to figure out, if anyone feels like they may be able to help, feel free to reach out.

This month marks an end to my Nix :heart: macOS sponsorship. I’m grateful for having had the opportunity to focus on Nixpkgs development for macOS. Going forward I won’t be able to spend as much time on this but I do want to finish the bump. If you’re short on time but want to help out the situation for Nix on macOS, consider contributing financially to the Open Collective.

Happy Holidays and a great 2023 to everyone!

20 Likes

Thank you for all the work you’ve done thus far! :heart:

3 Likes

Absolutely great, thanks so much for what you have done and the reports!

1 Like

Looks like you forgot to finish your sentence :slight_smile:

1 Like

I added a my contribution on open collective. It’s not much but a little bit per month will surely help :heart:

5 Likes
Hosted by Flying Circus.