Older pinned Nixpkgs fail with macOS 10.14 due to impure dependencies

Help
1

On a fresh install of macOS 10.14 Mojave some of our not-so-old projects (early/mid 2018) don’t run anymore. I think that this is due to these projects pinning a version of Nixpkgs that doesn’t have a notion of the new macOS yet. Updating the pinned version of Nixpkgs resolves the problem and the projects run again.

I understand that the root of this problem is that there are some impure packages that depend on files outside of Nix’s control (namely XCode libraries). My questions about this are:

  1. How can I know that impure packages have been introduced?
  2. Can I prevent such dependencies? Should I?
  3. What does this mean for older projects where I rely on a pinned Nix version from a few years back to get a specific set of dependencies: Is this not possible if there are impure dependencies in the project?

I’m asking to deepen my understanding of the problem and to see whether this could mean trouble further down the road, e.g. when we have to make changes to older projects in the future and need specific package versions.

The error message I got:

these derivations will be built:
  /nix/store/1d8c0ndga67wkwhbc23cb815hm3qr0p3-yarn-1.3.2.drv
building '/nix/store/1d8c0ndga67wkwhbc23cb815hm3qr0p3-yarn-1.3.2.drv'...
dyld: Library not loaded: /usr/lib/system/libsystem_network.dylib
  Referenced from: /nix/store/kxmwjyihrm04fdwbzin79406ab5vdp6v-Libsystem-osx-10.11.6/lib/libSystem.B.dylib
  Reason: image not found
builder for '/nix/store/1d8c0ndga67wkwhbc23cb815hm3qr0p3-yarn-1.3.2.drv' failed due to signal 6 (Abort trap: 6)
error: build of '/nix/store/1d8c0ndga67wkwhbc23cb815hm3qr0p3-yarn-1.3.2.drv' failed
1 Like
2

If you can update to latest 18.03, then the fix is available. Usually we don’t provide fixes for older than that but maybe this could be an exception? I know it’s a pretty significant issue on Mojave. As a side note if you can keep off of Mojave for longer, I would recommend it. Apple has yet to release some of the significant updates they made to the toolchain in 10.14, so we are going to have some minor issues until then (Getting unknown ld command 8 in Mojave · Issue #60 · tpoechtrager/cctools-port · GitHub).

Alternatively, if you maintain your own Nixpkgs fork, you could cherry pick from these three commits:

https://github.com/NixOS/nixpkgs/pull/43140

They are fairly self-contained so should be okay for 17.09 at least (not sure about 17.03 though).

2 Likes
3

Thanks for your reply, @matthewbauer! What I take away from this is that it’s easiest to just point to a newer version of Nixpkgs that has these fixes. And that I could – should I have to – fork the Nixpkgs repo at the point in time I require it to be and apply these patches. That fits my mental model and makes sense :slight_smile:

Just to clarify: when you say 18.03, are you talking about NixOS or Nixpkgs? Or are they, as I now assume, one and the same thing version-wise, because they’re so closely tied together? (I have always thought of NixOS to be a separate thing.)

As a last thing, maybe someone could still help me better understand my questions 1 and 2. The way I understand it now, after having seen the commits you referenced, all of Nixpkgs is closely tied to Apple’s libraries that get installed through their developer tools, so it’s not the case that a package could chose not to rely on those? I’m asking because I’ve seen that there’s an xcodebuild package, but then read somewhere (in an issue?) that it’s not well supported. I hoped that by carefully chosing the libraries I use to not use Apple’s libraries I could prevent such an issue in the future. Are these licencing issues or is it just too much work to become independent of Apple’s libraries? Or is this not possible at all because the OS is, after all, too important a part of this whole environment?

4

They are in the same repo so it can be nice to use it as a version. I think originally everyone just used master for Nixpkgs, but that can lead to a lot of headaches as things are not always thoroughly tested. I’ve been pushing for people to think of Nixpkgs in terms of version more often (although I don’t know if everyone does this).

It actually isn’t related to XCode or Command Line Tools. We don’t use either of those in Nixpkgs directly. You don’t need to have either of those installed to use Nixpkgs (in fact it’s recommended that you don’t as to avoid the impurity). The xcodebuild package actually uses Facebook’s xcbuild project (GitHub - facebookarchive/xcbuild: Xcode-compatible build tool.). It aims to be compatible with the Apple’s xcodebuild while also being built on open source.

The issue comes from Libsystem, which is Apple’s libc. This part of Nixpkgs is “impure”, but it comes with every install of macOS so it shouldn’t cause tons of issues. We rely on the Libsystem providing some compatibility for older versions of macOS. The problem is that Apple tends to restructure things a lot by moving libraries around. For this one, we were actually able to fix things very quickly (about 2 months before Mojave was released), but if you are on a pinned Nixpkgs commit, you will run into issues. If possible, I would recommend switching from package pinning to a specific commit and instead using the “channel” branches and updating frequently. Not only will this fix the Mojave issue but also make sure you receive security updates for up to a year (and we are trying to extend that window until a little longer).

5

BTW, we have the nixpkgs-*-darwin channels: Channels for NixOS project(s)

6

Thank you two, this is all very helpful! I’ll quickly summarize my understanding (please correct me if I’m wrong!) and present how I solved my issue at the end.

  1. Pinning Nixpkgs master is not a great idea because it’s a) not well tested, and b) doesn’t get any security/maintenance updates.
  2. Use channels instead, there are channels for Darwin too now: Channels for NixOS project(s)
  3. NixOS is a specialization of Nixpkgs and therefore it can help to think of the two as closely related and talk about them in terms of version numbers.
  4. Neither XCode or Command Line Tools are needed by Nixpkgs directly, it’s recommended to not have them installed to avoid accidental impurity.
  5. The Libsystem (Apple’s libc) is relied on heavily, though, and this is fine. Problems arise if Apple moves libraries around, but Nixpkgs is updated to reflect the new locations (see 2.)
  6. If there is a need to reference a very old version of Nixpkgs to get an old project running on a new machine, issues will likely arise in the Libsystem. Create a fork of that old version and apply patches to the Libsystem.

:white_check_mark: New solution
This is what I came up with for our shell.nix by following the ideas in this thread. It’s simpler and safer :slight_smile:

{ pkgs ? import (fetchTarball http://nixos.org/channels/nixpkgs-18.09-darwin/nixexprs.tar.xz) {} }:

let
  nodejs = pkgs.nodejs-8_x;
  yarn = pkgs.yarn.override { inherit nodejs; };

in pkgs.mkShell rec {
  buildInputs = [
    nodejs
    yarn
  ];
}

:x:Old solution
Here’s what I had before for reference (avoid doing this!):

let
  unpinned = import <nixpkgs> {};
in
{ pkgs ? import (unpinned.fetchFromGitHub
  { owner = "NixOS";
    repo = "nixpkgs";
    rev = "c26b40a933ea2b7fa4ca1fdc6900d54d726e73ff";
    sha256 = "1j1f0n7rl8vq91af9hrf98f98f449yr5i0p540q8gbsr504qid8b";
  }) {}
}:

let
  nodejs = pkgs.nodejs-8_x;
  yarn = pkgs.yarn.override { inherit nodejs; };

in pkgs.mkShell rec {
  buildInputs = [
    nodejs
    yarn
  ];
}
1 Like
7

(1) At least on x86_64-linux, master and the lagging “unstable” channel tend to work quite well, but these change relatively often in some potentially incompatible ways (and stable branches are usually a little more reliable, too). All security fixes go to master as well, though sometimes a little slower than to the active stable branch(es).

(3) The OS code is in the nixpkgs repository, so the commit numbers, release numbers etc. are referring to “the same thing”.

2 Likes