Nix 🖤 macOS Monthly

Hi, I’m working on macOS support in Nixpkgs, sponsored by the
Nix :black_heart: macOS Open Collective.
I’m a long-time macOS user and started using Nix almost 4 years ago.

This thread’s intended to be a place for monthly status updates on what
I’ve been working on and community feedback of painpoints macOS users
experience.

March 2021

I started tracking the state of macOS in issue
#116341 Status of macOS.
It’s a convenient place to keep track of known issues. I try to keep it
up to date on an overview level.

Nixpkgs currently uses macOS SDK version 10.12. This is fairly old and it
is starting to generate issues where newer software can’t be built for
macOS because it relies on features in newer SDKs. The first goal is to
get this updated. I’ve been working on
bumping it to 10.13,
the last version my local machine supports, as a way to familiarize myself
with the process. This should make it easier to update to at least 10.14,
the oldest version still supported by Apple, and likely newer versions using
cloud resources. I’ve gotten the first three stages of the Darwin stdenv
bootstrap to build (there’s 4 bootstrap stages and 1 final stage).

51 Likes

April 2021

I started out updating parts of the bootstrapFiles manually whenever I ran into a new problem that required newer tools or libraries. This became harder and harder to do because changes to one part of the bootstrapFiles required changes to other parts and it got messy. So the first thing I focused on was generating the bootstrapFiles properly using make-bootstrap-tools.nix. With some changes so I can generate new bootstrapFiles while passing in new bootstrapFiles.

I’ve also started updating the LLVM version used for Darwin, both the stdenv and the default for packages. The PR can be found here. This is long overdue and has held back the default LLVM version even on Linux (though voices have been raised to change this in 21.05). I’m hoping this solves a build issue involving Swift CoreFoundation and macOS version 10.13.

Some exciting news is that @abathur’s work on improving the installer has been merged :tada:
They’re currently looking for someone to pick up the project of completing full uninstall/reinstall support as they’re short on time. (I believe this wouldn’t be specific to the macOS installer.)

12 Likes

Correct! The initial focus would be the multi-user installer. How to handle single-user is an open question. :slight_smile:

2 Likes

May 2021

This month was spent getting the bootstrap-tools and stdenv to build with LLVM 11. This required quite a few patches essentially flipping clang switches back to the clang-7 behavior, like -fcommon.
I based this work on staging, which got the Apple Silicon PR merged in time for the 21.05 release, in an attempt to make it in the release as well but got stuck on two regressions introduced in that PR.
These were quickly patched by @thefloweringash :heart:.
On @LnL7’s advice I’ve rebased the LLVM bump PR on a specific Nixpkgs master commit so we can have a hydra jobset that compares well with existing evaluations and finish up this work.

In other news, the Apple Silicon PR was merged in time for release 21.05 :tada:.
This means we’re getting really close to officially supporting Aarch64 macs, pending some fixes to enable Aarch64 hydra builds and a Nix release.
There has also been progress on getting NixOS tests to run virtualized on Darwin by r2r-dev.

19 Likes

I think that’s done now, at least for the short term. NixPkgs branches covered:

1 Like

Nix stable release (for aarch64-darwin) is worked on here, but it seems to require a change to the nix jobset in hydra.

This is great! Thanks for posting these monthly updates.

2 Likes

June 2021

The Hydra build of the LLVM bump progressed very slowly because the minis were swamped by a couple staging rebuilds, so I focused on the SDK bump. I rebased the previous work on the LLVM bump and started updating the Apple source releases. The source releases are what Nixpkgs tries to depend on for macOS builds when possible. Updating them involves adding to the header lists the releases are checked against when new headers are added, as for CommonCrypto this time around. Sometimes headers change location in the source tarball, requiring an update to the buildPhase, like for the new libdispatch. And occasionally headers are removed in new version, in this case we create a package that combines headers from the new and old version so code that relies on these headers still compiles, as happened to libdispatch and xnu.

I looked into building packages against newer SDK frameworks, which came up in this issue. But I got stuck on the rewriting of dylib paths and didn’t see a clear way forward. Finally with most of the Hydra build for the LLVM bump finished I was able to start looking into fixing the cups build which caused most of the new build failures due to being a common (transitive) dependency, I have a patch but ended up running into failures the hydra builders haven’t encountered and am looking into this. The new Hydra evaluation has started.

I also wrote a tutorial to get people started on recent versions of macOS (soon regardless of CPU), with step-by-step uninstallation instructions to reduce the fear of ending up with a cluttered system or make it easy to start from scratch. This is intended to go into the Nix manual. A nix-darwin setup tutorial would be next.

8 Likes

July 2021

This month was spent focusing on the LLVM bump. The latest Hydra Evaluation is promising. The most common build failure in this latest evaluation is icoutils and fixing it required a patch to Libc which has revealed an impurity in the build for libpulseaudio, two steps forward, one step back. But we’re getting closer to an LLVM 11 stdenv on Darwin. If there’s packages in the evaluation that are failing to build but are important to you, patches are welcome.

I also looked into why Firefox fails to build on Darwin. First stumbling block is jemalloc and I’m talking to maintainers. So as a temporary workaround I’ve uploaded the expression I use, which fetches the latest app in a dmg from mozilla.org, to NUR. Making use of this should be as simple as putting an override in your config:

{
  nixpkgs.config.packageOverrides = pkgs: {
    nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") {
      inherit pkgs;
    };
  };
}

And then you can use nur.repos.toonn.apps.firefox to install the package. (For setups without nix-darwin take a look at the NUR installation instructions.) The expression has a version argument so it’s possible to install a specific Firefox version, every version since 86.0, like so, nur.repos.toonn.apps.firefox.override { version = "86.0"; }.

10 Likes

August 2021

At the start of the month I finished up some work on the LLVM bump. I patched Libc to define TARGET_OS_EMBEDDED if it is not defined because LLVM 11 is stricter about the use of undefined identifiers. I also updated the libpulseaudio expression to build without impurities. This should take care of most of the major breakages introduced by the LLVM update. I’m currently waiting for a Hydra evaluation to confirm this but the x86_64 Darwin builders are swamped by a stdenv rebuild due to an OpenSSL update and Haskell packages updates.

Most of my time was spent on troubleshooting the configd build, still working on this branch though I’ve been going back and forth on changes recently, have yet to distill them into commits to push. Updating the Apple source releases is an integral part of bumping the SDK version, in Nixpkgs they are an amalgam of packages from different version of macOS, our XNU has header from two versions of XNU, for example, and Security and configd haven’t gotten updated for several major versions of macOS. Setting out to update all of them to the versions available for macOS 10.13.6 I quickly ran into stumbling blocks. Some packages have changed considerably, in the case of hfs I decided not to pursue updating it now because it would be too time consuming, most of the old headers simply aren’t present in the newer release and it’s not clear what it does provide now. Configd has also proven to be rather difficult, it uses functions that are missing unless -DPRIVATE or -DKERNEL are provided but neither of these seems to be intended for third-party builds, the Makefile only provides -D__OPEN_SOURCE__. With both those flags configd starts depending on headers from Neon, a C HTTP/WebDAV client library implementation, which is available on opensource.apple.com but we can only guess at the version that corresponds to a certain version of macOS.

This means the source releases Apple provides aren’t necessarily even buildable and I’ll probably hold off on updating configd as a consequence. During this troubleshooting I did find out the AvailabilityInternal.h header Apple provides in the XNU sources is no longer updated since macOS 10.12.6. Systems running later versions of macOS do have an updated header locally but the open source release is missing several definitions used by other open source releases. I originally tried fixing this by patching the missing definitions into the header, which could be done programmatically, but I quickly realized why Apple changed to a different definition for availability macros, the old approach leads to a combinatorial explosion of definitions for all combinations of macOS versions, the header on my system has 10k more lines than the header in the open source release and this difference would only get bigger with newer versions. So instead I opted to update the relatively fewer places where the old-style availability macros were used for newer versions to using the newer macros. This made it possible to update Security, which was at the version corresponding to macOS 10.9.5 with a note stating it was to be updated as soon as the problems were figured out, 4 major versions of macOS later I think we did it : )

18 Likes

September 2021

The last issue for the LLVM bump is I had patched two packages, wheel and sphinx, because of a hash mismatch. I figured the tarballs had simply changed and required new hashes but changing the hash caused a similar hash mismatch on Linux. Turns out this problem was due to the file systems involved, which symphorien and VladimirCunat helped me realise. On Linux unicode is usually encoded using UTF-8 and normal form C (NFC). The HFS+ file system, which was the default on macOS until APFS replaced it in High Sierra (10.13), uses UTF-16 and normal form D (NFD). This means hashing the same file name across these file systems can have different results. Both these python packages contain tests using filenames with characters that are represented differently in NFC and NFD, so I patched their sources to use normalization-resistant characters, this seems to have worked fine for wheel, but not for sphinx, looking into it though. Ideally upstream will be interested in these changes otherwise we’ll have to carry the patches to guarantee these packages are FODs. A more general solution would be to change the hashing Nix does to normalize all unicode in a fixed way. Not sure this is a path we want to walk though, there are strings which do not round-trip through normal forms so it’s not 100% guaranteed to work.

As for the SDK bump, configd remains a hard nut to crack. Somewhere around macOS 10.13 Apple has stopped releasing all the headers that are necessary to build it. XPC no longer seems avoidable as a dependency, CoreFoundation seems incomplete (we rely on Darling for the missing bits now), there’s some packages hosted on opensource.apple.com but not listed in the macOS releases, like neon and OpenBSM. I don’t see an alternative to getting the XPC headers from the SDK and likewise for some other missing headers. There’s several projects online where missing headers are worked around by stubbing, like OSXPrivateSDK and GoVPN, but it doesn’t seem like a good way to deal with the issue. Fabricating a constant can lead to unexpected behavior. That’s why I intend to use binaries from the SDK whenever dependencies aren’t available. This will be a step back with regards to building from source unfortunately.

13 Likes