Link errors on darwin caused by darwin.CF

elaforge · January 27, 2022, 8:18am

Linking a GUI-using app in a nix-shell, via ghc, I suddenly started getting these errors:

Undefined symbols for architecture x86_64:
"_OBJC_CLASS_$_NSData", referenced from:
   objc-class-ref in libfltk.a(Fl_MacOS_Sys_Menu_Bar.o)
"_NSDefaultRunLoopMode", referenced from:
   Fl_Cocoa_Screen_Driver::ready() in libfltk.a(Fl_cocoa.o)
   Fl_Cocoa_Screen_Driver::wait(double) in libfltk.a(Fl_cocoa.o)
   Fl_Cocoa_Screen_Driver::open_display_platform() in libfltk.a(Fl_cocoa.o)
[ etc ]

After a bunch of poking about with verbose flags, the problem seems to be that swift-corefoundation (from darwin.CF) winds up as a -F flag. Actually very many -F flags show up, many of them repeated, which I assume is down to using bash for stdenv and is usually harmless… except when it’s not. I wind up with both the swift-corefoundation and /nix/store/xyz-apple-framework-CoreFoundation (from darwin.cf-private). It seems to work if I have neither -F flags, or if I have both but but apple-framework comes last. If I have just swift, or swift comes after apple-framework, then link failure. They both have a CoreFoundation.framework, so presumably you don’t want to find the wrong one… or either of them? Strange. To be clear, I’m not doing anything to do with swift, it’s just along for the ride for some reason.

When trying to track down how it winds up on the ghc link line, it turns out it’s always there regardless of any env vars, and I suspect it’s because all hackage package.conf files have acquired a framework-dirs: /nix/store/blahblah-swift-etc. I assume this is somehow due to it winding up in some env var, perhaps due to it going in defaultBuildInputs, which seems to be undocumented and I can’t quite trace where it goes.

This all seems to be related to rstudio: issues with darwin frameworks · Issue #24237 · NixOS/nixpkgs · GitHub, and I must have stumbled across this a while back because I had the -F/System/Library/Frameworks hack in there… and now after updating nixpkgs it’s no longer working. There are about a zillion flags on and being in the wrong order is fatal so no surprise if it’s brittle.

Anyway, my thought is to try to get that swift thing out of framework-dirs, but before I put another evening into trying to see how it gets in there… does anyone have any clues? Is there a better way to debug this? Why are there multiple CoreFoundations? What is each one for? Why would I not have the link error if I don’t link any of them, but get it if I have both but in the wrong order? Also, this is just about -F flags, which is about the paths… I’m not directly linking CoreFoundation. However, I do link Cocoa and maybe they pull each other in. What is defaultBuildInputs, and what effect does it have? How does an apparently swift library wind up in all haskell packages?

Oh, and this is intel, not aarch64, if it makes a difference.

Thanks in advance!

elaforge · February 6, 2022, 5:44am

Not sure if anyone is interested, but in case someone later has the same problem, I eventually figured this out by explicitly passing -framework-path=${darwin.cf-private}/Library/Frameworks to ghc. That seems to be enough to get it to stop tripping over the swift one. I still don’t understand what’s going on, but at least it links.

One other critical thing that I forgot (even after taking notes! but in the wrong spot), was that the whole thing started when I tried to use clang from nixpkgs instead of tho OS X “commandline tools” one. Presumably the native system one is looking in system places and this was enough to not get fooled by the swift corefoundation… or rather it would obey the explicit -F/System/LibraryFrameworks while the nix clang wouldn’t? I don’t know… which is why I expect to be here again in a year or two, and maybe this will at least be another note to myself, if I some lose all the others!

veprbl · February 6, 2022, 8:38am

See

github.com

NixOS/nixpkgs/blob/bb96e481b317c770281c1bfd9e6face9bfe0888f/pkgs/top-level/darwin-packages.nix#L171-L183

    
      
          # This attribute (CF) is included in extraBuildInputs in the stdenv. This
          # is typically the open source project. When a project refers to
          # "CoreFoundation" it has an extra setup hook to force impure system
          # CoreFoundation into the link step.
          #
          # In this branch, we only have a single "CoreFoundation" to choose from.
          # To be compatible with the existing convention, we define
          # CoreFoundation with the setup hook, and CF as the same package but
          # with the setup hook removed.
          #
          # This may seem unimportant, but without it packages (e.g., bacula) will
          # fail with linker errors referring ___CFConstantStringClassReference.
          # It's not clear to me why some packages need this extra setup.

Whenever you depend on a closed source framework like darwin.apple_sdk.frameworks.Cocoa, it will pull in darwin.apple_sdk.frameworks.CoreFoundation to replace the incompatible but opensource darwin.CF.

elaforge · February 6, 2022, 9:38am

Hmm I think I must have stumbled across that comment when I went through that file, but didn’t understand what it was trying to say. Actually I still don’t really, though your additional comment makes me think that depending on Cocoa is intended to automatically replace open source darwin.CF (is that the one that has swift in its name?) with the impure Apple one? In that case, that particular magic seems to not be working with ghc… so perhaps my manually adding the impure one has accidentally fixed that?

If so, one guess about why it doesn’t work with ghc is that ghc’s mkDerivation unconditionally hardcodes the swift CF into link flags propagated by every library, which means the link line winds up with both, which means it depends on the order, which is fragile. I don’t know why mkDerivation does that, but it’s intentional so there must have been some reason.

veprbl · February 13, 2022, 9:36am

Yes, a setup hook

github.com

NixOS/nixpkgs/blob/8436842aaf7f3b6bbe7fe2d755ab0d6cc75155bf/pkgs/os-specific/darwin/apple-sdk/cf-setup-hook.sh

linkSystemCoreFoundationFramework() {
  NIX_CFLAGS_COMPILE="-F@out@/Library/Frameworks${NIX_CFLAGS_COMPILE:+ }${NIX_CFLAGS_COMPILE-}"
  # gross! many symbols (such as _OBJC_CLASS_$_NSArray) are defined in system CF, but not
  # in the opensource release
  # if the package needs private headers, we assume they also want to link with system CF
  NIX_LDFLAGS+=" @out@/Library/Frameworks/CoreFoundation.framework/CoreFoundation"
}

preConfigureHooks+=(linkSystemCoreFoundationFramework)

is enabled when darwin.apple_sdk.frameworks.CoreFoundation is used

github.com

NixOS/nixpkgs/blob/8436842aaf7f3b6bbe7fe2d755ab0d6cc75155bf/pkgs/os-specific/darwin/apple-sdk/default.nix#L283-L285

    
      
          CoreFoundation = lib.overrideDerivation super.CoreFoundation (drv: {
            setupHook = ./cf-setup-hook.sh;
          });

It seems like Cocoa depends on AppKit, which depends on Foundation, which depends on CoreFoundation: https://github.com/NixOS/nixpkgs/blob/8436842aaf7f3b6bbe7fe2d755ab0d6cc75155bf/pkgs/os-specific/darwin/apple-sdk/frameworks.nix

It would appear that the cf-setup-hook.sh prepends the -F flag, so they should always come first. If that is not the case, it could be because of a bug. You could file an issue to https://github.com/NixOS/nixpkgs/issues. If you do, please, include a minimal reproducing example.