MacOS System Integrity Protection and {DY,}DL_LIBRARY_PATH

Help
1

It seems that Apple’s System Integrity Protection (SIP) causes (among other restrictions) {DY,}LD_LIBRARY_PATH to be ignored in many circumstances.

Geant4 is a framework which clients use by writing their own code which depends on the Geant4 libraries, being expected to specify the location of these libraries by setting DL_LIBRARY_PATH.

nixpgks.geant4 works like a charm on Linux, but gives mysterious crashes on Darwin. When using this derivation on Linux, DL_LIBRARY_PATH contains locations of Geant4 in the nix store; when using it on
MacOS, {DY,}DL_LIBRARY_PATH are empty.

As I am not an apple user, and do not have interactive access to a MacOS machine, getting to the bottom of this is difficult. Specifically, I do not know under exactly what conditions these environment variables are empty; I do vaguely recall some evidence of their being set correctly in some circumstances. I suspect that this is related to the exact circumstances under which SIP sanitizes these PATHs.

Besides completely disabling SIP on the system (something which users would reasonably decline to do), what can be done make the nixpgs.geant4 usable on MacOS?

2

I’m unsure from your description when/where the envs are set and when/where they are empty.

Someone on Matrix pointed out this Apple doc: Runtime Protections.

It says:

Spawning children processes of processes restricted by System Integrity Protection, such as by launching a helper process in a bundle with NSTask or calling the exec(2) command, resets the Mach special ports of that child process. Any dynamic linker (dyld ) environment variables, such as DYLD_LIBRARY_PATH , are purged when launching protected processes.

Do you know if there’s a system executable that might be in the execution chain via PATH or abspath?

Same person on IRC also pointed out an issue suggesting that the upstream may not be using DLYD_LIBRARY_PATH any more? https://bugzilla-geant4.kek.jp/show_bug.cgi?id=1960

3

Given a flake which contains a devShell which has geant4 and geant4.data.{all,of,them} in packages, in the shell created by nix develop: LD_LIBRARY_PATH is set on Linux, but empty on macOS (as is DYLD_LIBRARY_PATH). However all of the envvars which point at the data directories (such as G4LEDATA) are set on both Linux and macOS.

IOW, the flake is setting G4-related envvars on both systems, but on macOS {DY,}LD_LIBRARY_PATH are empty.

Here is

PATH 
/nix/store/ymixmqqrdi26w5b206y73f23k8bbn528-geant4-11.0.4/bin:
/nix/store/abnjrd63vr1qmbqfz234kzpnck25bhyk-clhep-2.4.6.4/bin:
/nix/store/sqnyvmbx2b3lzpakd4v6bm0b6h8arndm-expat-2.5.0-dev/bin:
/nix/store/hl0k2pxfv06592vzkw5a7w1nravk5szv-xerces-c-3.2.4/bin:
/nix/store/nd510lm77l688m424qx97wj7l7n8xml3-qtbase-5.15.9-dev/bin:
/nix/store/b8w8gwvcnnz836na970xiah9pl5xlaqq-libxml2-2.10.4-dev/bin:
/nix/store/202ngv2cq0zakilkq6vj8i58r71p38mb-libiconv-50/bin:
/nix/store/ljqrv5is3ggl58h5hn5jcl3kpd02117y-libxml2-2.10.4-bin/bin:
/nix/store/dqrrziv0wmjkghs3cxfq65r054hfv2yy-libxslt-1.1.37-dev/bin:
/nix/store/b5xhhsyn8hwmxp9knrfg61hsazik73c3-libxslt-1.1.37-bin/bin:
/nix/store/f9h9d7dvwzc8m03qg2kd9sq354igvzp3-openssl-3.0.9-bin/bin:
/nix/store/ply4nxpzl1smn4zdy6zyvbngyj7cgsnc-sqlite-3.41.2-bin/bin:
/nix/store/2v05iq73yfnzjk9hqqv7s448gi03127c-harfbuzz-7.2.0-dev/bin:
/nix/store/c9zx8jp93jsr9vh3xl5jgsrgs90k3v2s-graphite2-1.3.14/bin:
/nix/store/kahaj756xqkj4lxi0yia32jna0vpw81q-icu4c-73.1-dev/bin:
/nix/store/r9vw23x8kqdnqz5p318m3s80wadia29n-libjpeg-turbo-2.1.5.1-bin/bin:
/nix/store/07s2blw77hkmysq59pcg5hdj2a8raxzn-libpng-apng-1.6.39-dev/bin:
/nix/store/43zm9vmfqy7bvw94ys5alp3a25s27rkc-pcre2-10.42-dev/bin:
/nix/store/xlq9kylbjvbvcb9z1q1c58lp2vanmq5k-pcre2-10.42-bin/bin:
/nix/store/456a1s0fb2f4n4j56b7qdbgcd0mhjc8n-clang-wrapper-16.0.1/bin:
/nix/store/fmckcvpqs18licyg25qyhd2my3av755q-clang-16.0.1/bin:
/nix/store/x85sf9jfs19xn8047dy0wybhl7p496wp-coreutils-9.1/bin:
/nix/store/0zyylnbr251v71n76cr4qbanq2rhlrm9-cctools-binutils-darwin-wrapper-973.0.1/bin:
/nix/store/2lc47n91bb2awx8680rbj5rphvvlg1af-cctools-binutils-darwin-973.0.1/bin:
/nix/store/wjs5hi0iysi6dbn2y4lrc9h2j1axqzki-clang-tools-14.0.6/bin:
/nix/store/c3xg4s49szaimvai226l0wycszyx8b45-cmake-3.25.3/bin:
/nix/store/mq8zr8a1cq0yhqpy8l33b247x831y0mf-ps-adv_cmds-119/bin:
/nix/store/69jh796v9wx506qfggv50j5r9aq6p80k-cmake-language-server-0.1.7/bin:
/nix/store/b82gpsyfms41hhrah9mnl9yzpxabkn67-cmake-format-0.6.13/bin:
/nix/store/j0d7gnlh4gp8qmd77yjgmx3yg7k5har2-python3.10-autopep8-2.0.1/bin:
/nix/store/4b73ryvi1xn64gpl532psdqyv8l61v5b-python3.10-pycodestyle-2.10.0/bin:
/nix/store/pdbh59881kadh4ai449f710lq30qd187-python3-3.10.11/bin:
/nix/store/863k01hkzi1w31j0h32sq753hyk7a5ql-python3.10-flake8-6.0.0/bin:
/nix/store/3iqkj45b38frnzff9r4g6rwsiwka5z3r-python3.10-pyflakes-3.0.1/bin:
/nix/store/zk2yg8gaqrp5czjwbm7pmji6qi487b49-python3.10-babel-2.12.1/bin:
/nix/store/ccg66ay40y43kwhi83761qx592q6xjnm-python3.10-pylint-2.16.2/bin:
/nix/store/l001hhp658ci63cmdghf28m8iszhzm8p-python3.10-dill-0.3.6/bin:
/nix/store/y78kbqnmlvrklv313acg0m537rhcp3p6-python3.10-isort-5.12.0/bin:
/nix/store/3aqsapyziv3lspkyb9z7criyhk77w35m-just-1.13.0/bin:
/nix/store/1l7ilm1wzqy6qqlf03kjspgkqmm216dr-gnused-4.9/bin:
/nix/store/dwfi5lsq55jjxy5bcljg67dkc87lrwyz-mdbook-0.4.28/bin:
/nix/store/ymixmqqrdi26w5b206y73f23k8bbn528-geant4-11.0.4/bin:
/nix/store/x85sf9jfs19xn8047dy0wybhl7p496wp-coreutils-9.1/bin:
/nix/store/l4abq84b589bp32pp4m71f1ans9dsx44-findutils-4.9.0/bin:
/nix/store/6nvjsdiy7hwg71cgnqqna2z2jw622zdb-diffutils-3.9/bin:
/nix/store/1l7ilm1wzqy6qqlf03kjspgkqmm216dr-gnused-4.9/bin:
/nix/store/qmik2dlswqhnc6r8gvrqza3rk88idalm-gnugrep-3.7/bin:
/nix/store/4hhs796cvzkrdfrdrqcc077ra4m6ir84-gawk-5.2.1/bin:
/nix/store/0zxcz4556v2ybkvnpjrzabfdwdywx88c-gnutar-1.34/bin:
/nix/store/w70vg29f8d3n3p11b2xnjkj543z339pf-gzip-1.12/bin:
/nix/store/sqr63fq94y864l45vgkq35y9slkk9pg4-bzip2-1.0.8-bin/bin:
/nix/store/9b2dfpv3w1jnp9fs4pnrphvwsdx4mhwx-gnumake-4.4.1/bin:
/nix/store/vj4xxsds1bkidpj89ma5f2msxf6mm4m3-bash-5.2-p15/bin:
/nix/store/vwjsnfvq67mf10rhyj5l6a8zs26v8xl7-patch-2.7.6/bin:
/nix/store/s3a12da912cl43p8xb4zyyqmaxg3qp5s-xz-5.4.3-bin/bin:
/nix/store/l3l6algkiwsq4mkxk7kkvf1ixvd3gxi1-file-5.44/bin:
/usr/local/lib/ruby/gems/3.0.0/bin:
/usr/local/opt/ruby@3.0/bin:
/usr/local/opt/pipx_bin:
/Users/runner/.cargo/bin:
/usr/local/opt/curl/bin:
/usr/local/bin:
/usr/local/sbin:
/Users/runner/bin:
/Users/runner/.yarn/bin:
/Users/runner/Library/Android/sdk/tools:
/Users/runner/Library/Android/sdk/platform-tools:
/Library/Frameworks/Python.framework/Versions/Current/bin:
/Library/Frameworks/Mono.framework/Versions/Current/Commands:
/Users/runner/.nix-profile/bin:
/nix/var/nix/profiles/default/bin:
/Users/runner/.nix-profile/bin:
/nix/var/nix/profiles/default/bin:
/usr/local/lib/ruby/gems/3.0.0/bin:
/usr/local/opt/ruby@3.0/bin:
/usr/local/opt/pipx_bin:
/Users/runner/.cargo/bin:
/usr/local/opt/curl/bin:
/usr/local/bin:
/usr/local/sbin:
/Users/runner/bin:
/Users/runner/.yarn/bin:
/Users/runner/Library/Android/sdk/tools:
/Users/runner/Library/Android/sdk/platform-tools:
/Library/Frameworks/Python.framework/Versions/Current/bin:
/Library/Frameworks/Mono.framework/Versions/Current/Commands:
/usr/bin:
/bin:
/usr/sbin:
/sbin:
/Users/runner/.dotnet/tools:
/Users/runner/.ghcup/bin:
/Users/runner/hostedtoolcache/stack/2.11.1/x64:
/Users/runner/.dotnet/tools:
/Users/runner/.ghcup/bin:
/Users/runner/hostedtoolcache/stack/2.11.1/x64

in nix develop on macOS GitHub Actions

… but now I notice there’s something else even weirder going on, which I thought we had sidestepped already, but apparently not … I need to dig a bit more.

Ugh. There are so many ways in which this can be broken on macOS. Ideally, I’d find a working example of a flake that successfully uses nixpkgs.geant4 on macOS, to make sure that I’m not barking up the wrong tree.

1 Like
4

We’ve reduced it to a minimal example.

  1. We can compile hello-world with plain clang++ and run the resulting executable on macOS.

  2. We can compile and run this same program with cmake.

  3. As soon as we add

    find_package(Geant4 REQUIRED)
include(${Geant4_USE_FILE})

    to CMakeLists.txt, our hello-world program segfaults, when trying to access the standard library:

    Process 7764 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
Target 0: (exampleB1) stopped.
(lldb) up
frame #1: 0x0000000102849bb8 libc++.1.0.dylib`std::__1::__stdinbuf<char>::imbue(std::__1::locale const&) + 52
libc++.1.0.dylib`std::__1::__stdinbuf<char>::imbue:
->  0x102849bb8 <+52>: str    w0, [x19, #0x58]
    0x102849bbc <+56>: ldr    x0, [x19, #0x48]
    0x102849bc0 <+60>: ldr    x8, [x0]
    0x102849bc4 <+64>: ldr    x8, [x8, #0x38]
    rest of stack, for context 
    (lldb) up
frame #2: 0x000000010284941c libc++.1.0.dylib`std::__1::DoIOSInit::DoIOSInit() + 148
libc++.1.0.dylib`std::__1::DoIOSInit::DoIOSInit:
->  0x10284941c <+148>: add    x0, sp, #0x10
    0x102849420 <+152>: bl     0x10284b034               ; std::__1::locale::~locale()
    0x102849424 <+156>: nop    
    0x102849428 <+160>: ldr    x8, #0x6ed40              ; (void *)0x00000001031743d0: vtable for std::__1::basic_istream<char, std::__1::char_traits<char> >
(lldb) 
frame #3: 0x000000010284a9d4 libc++.1.0.dylib`_GLOBAL__I_000100 + 72
libc++.1.0.dylib`:
->  0x10284a9d4 <+72>: adr    x0, #-0xf58               ; std::__1::DoIOSInit::~DoIOSInit()
    0x10284a9d8 <+76>: nop    
    0x10284a9dc <+80>: adr    x2, #-0x369dc
    0x10284a9e0 <+84>: nop    
(lldb) 
frame #4: 0x00000001a0b4c1e0 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const:
->  0x1a0b4c1e0 <+168>: add    x0, sp, #0x10
    0x1a0b4c1e4 <+172>: bl     0x1a0b2ba90               ; dyld3::ScopedTimer::endTimer()
    0x1a0b4c1e8 <+176>: ldp    x29, x30, [sp, #0xa0]
    0x1a0b4c1ec <+180>: ldp    x20, x19, [sp, #0x90]
(lldb) 
frame #5: 0x00000001a0b8dc60 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 172
dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const:
->  0x1a0b8dc60 <+172>: add    x21, x21, #0x8
    0x1a0b8dc64 <+176>: cmp    x21, x22
    0x1a0b8dc68 <+180>: b.lo   0x1a0b8dbf0               ; <+60>
    0x1a0b8dc6c <+184>: b      0x1a0b8dd2c               ; <+376>
(lldb) 
frame #6: 0x00000001a0b811a4 dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528
dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const:
->  0x1a0b811a4 <+528>: ldrb   w8, [x19]
    0x1a0b811a8 <+532>: cbnz   w8, 0x1a0b81308           ; <+884>
    0x1a0b811ac <+536>: add    x8, x25, #0x50
    0x1a0b811b0 <+540>: cmp    x25, x23
(lldb) 
frame #7: 0x00000001a0b2c2d8 dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
dyld`dyld3::MachOFile::forEachLoadCommand:
->  0x1a0b2c2d8 <+296>: ldrb   w8, [sp, #0x3f]
    0x1a0b2c2dc <+300>: cbnz   w8, 0x1a0b2c25c           ; <+172>
    0x1a0b2c2e0 <+304>: add    w22, w22, #0x1
    0x1a0b2c2e4 <+308>: ldr    w8, [x20, #0x10]
(lldb) 
frame #8: 0x00000001a0b801cc dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
dyld`dyld3::MachOFile::forEachSection:
->  0x1a0b801cc <+192>: sub    x0, x29, #0x38
    0x1a0b801d0 <+196>: bl     0x1a0b2c354               ; Diagnostics::assertNoError() const
    0x1a0b801d4 <+200>: add    x0, sp, #0x48
    0x1a0b801d8 <+204>: mov    w1, #0x8
(lldb) 
frame #9: 0x00000001a0b82cfc dyld`dyld3::MachOFile::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, bool&) block_pointer) const + 160
dyld`dyld3::MachOFile::forEachInitializerPointerSection:
->  0x1a0b82cfc <+160>: ldp    x29, x30, [sp, #0x60]
    0x1a0b82d00 <+164>: ldp    x20, x19, [sp, #0x50]
    0x1a0b82d04 <+168>: ldp    x22, x21, [sp, #0x40]
    0x1a0b82d08 <+172>: add    sp, sp, #0x70
(lldb) 
frame #10: 0x00000001a0b8d904 dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432
dyld`dyld3::MachOAnalyzer::forEachInitializer:
->  0x1a0b8d904 <+432>: mov    x8, sp
    0x1a0b8d908 <+436>: adrp   x16, 374721
    0x1a0b8d90c <+440>: add    x16, x16, #0xed8          ; _NSConcreteStackBlock
    0x1a0b8d910 <+444>: mov    x17, x8
(lldb) 
frame #11: 0x00000001a0b48864 dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 448
dyld`dyld4::Loader::findAndRunAllInitializers:
->  0x1a0b48864 <+448>: ldrb   w8, [x21, #0x21]
    0x1a0b48868 <+452>: cbz    w8, 0x1a0b48924           ; <+640>
    0x1a0b4886c <+456>: add    x8, sp, #0x68
    0x1a0b48870 <+460>: add    x8, x8, #0x8
(lldb) 
frame #12: 0x00000001a0b48c18 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
dyld`dyld4::Loader::runInitializersBottomUp:
->  0x1a0b48c18 <+220>: ldp    x29, x30, [sp, #0x40]
    0x1a0b48c1c <+224>: ldp    x20, x19, [sp, #0x30]
    0x1a0b48c20 <+228>: ldp    x22, x21, [sp, #0x20]
    0x1a0b48c24 <+232>: ldp    x24, x23, [sp, #0x10]
(lldb) 
frame #13: 0x00000001a0b48bf4 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
dyld`dyld4::Loader::runInitializersBottomUp:
->  0x1a0b48bf4 <+184>: add    w23, w23, #0x1
    0x1a0b48bf8 <+188>: cmp    w23, w22
    0x1a0b48bfc <+192>: b.ne   0x1a0b48b80               ; <+68>
    0x1a0b48c00 <+196>: mov    x0, x19
(lldb) 
frame #14: 0x00000001a0b4c26c dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const:
->  0x1a0b4c26c <+112>: ldr    x1, [x19]
    0x1a0b4c270 <+116>: ldr    x8, [x1, #0x30]
    0x1a0b4c274 <+120>: lsl    x12, x8, #3
    0x1a0b4c278 <+124>: add    x9, x12, #0x8
(lldb) 
frame #15: 0x00000001a0b48d98 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks:
->  0x1a0b48d98 <+304>: ldrb   w8, [x19, #0x21]
    0x1a0b48d9c <+308>: cbz    w8, 0x1a0b48e58           ; <+496>
    0x1a0b48da0 <+312>: add    x8, sp, #0x8
    0x1a0b48da4 <+316>: add    x8, x8, #0x8
(lldb) 
frame #16: 0x00000001a0b6c984 dyld`dyld4::APIs::runAllInitializersForMain() + 468
dyld`dyld4::APIs::runAllInitializersForMain:
->  0x1a0b6c984 <+468>: mov    x0, x20
    0x1a0b6c988 <+472>: mov    x1, x19
    0x1a0b6c98c <+476>: bl     0x1a0b44b04               ; dyld4::Loader::analyzer(dyld4::RuntimeState&) const
    0x1a0b6c990 <+480>: bl     0x1a0b7fea0               ; dyld3::MachOFile::isMainExecutable() const
(lldb) 
frame #17: 0x00000001a0b312d0 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3480
dyld`dyld4::prepare:
->  0x1a0b312d0 <+3480>: bl     0x1a0b6aa7c               ; dyld4::notifyMonitoringDyldMain()
    0x1a0b312d4 <+3484>: mov    w0, #0x4
    0x1a0b312d8 <+3488>: movk   w0, #0x1f07, lsl #16
    0x1a0b312dc <+3492>: bl     0x1a0b2bdc0               ; dyld3::kdebug_trace_dyld_enabled(unsigned int)
(lldb) 
frame #18: 0x00000001a0b2fe18 dyld`start + 1964
dyld`start:
->  0x1a0b2fe18 <+1964>: mov    x19, x0
    0x1a0b2fe1c <+1968>: ldrb   w8, [sp, #0xb1]
    0x1a0b2fe20 <+1972>: cbnz   w8, 0x1a0b2fe88           ; <+2076>
    0x1a0b2fe24 <+1976>: ldrb   w8, [sp, #0xb0]
(lldb) 
error: Already at the top of the stack.

    This stack trace comes from a machine running Apple Silicon, but we get essentially the same on an Intel machine. (No problems whatsoever on Linux.)

So it seems that merely adding the (nix flake-provided) Geant4 libraries as dependencies to the hello-world executable, prevents the latter from being able to find the standard libraries at run-time.

1 Like
5

Is that the flake being used? When I run the example, I get a segmentation fault. Based on your debugger trace, I knew what it was.

$ otool -L ./g4-examples/B1/build/exampleB1 | rg 'libc++|Qt'
	/nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5OpenGL.5.dylib (compatibility version 5.15.0, current version 5.15.9)
	/nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5PrintSupport.5.dylib (compatibility version 5.15.0, current version 5.15.9)
	/nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5Widgets.5.dylib (compatibility version 5.15.0, current version 5.15.9)
	/nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5Gui.5.dylib (compatibility version 5.15.0, current version 5.15.9)
	/nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5Core.5.dylib (compatibility version 5.15.0, current version 5.15.9)
	/nix/store/1wd5dggwpcnrjjlfjykl6hf2x03v6q6i-libcxxabi-16.0.1/lib/libc++abi.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/nix/store/mwpcf9kxw8k7w6wx9x3qjrj2rmjcmrhc-libcxx-16.0.1/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)
$ otool -L /nix/store/zca5zd4mc9121a3r4sarm03gb3xwy4bd-qtbase-5.15.9/lib/libQt5Core.5.dylib | rg 'libc++'
	/nix/store/ldjmd9ry137lb5f4s7lklp6scij0j9b5-libcxxabi-11.1.0/lib/libc++abi.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/nix/store/29dxsv48zqzb6w65jdb0agjhryhl0vyx-libcxx-11.1.0/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)

You’re linking two versions of libc++ into the same binary (Qt and geant4 are both linking against libc++ 11 while the example is linking against libc++ 16). That’s what is causing the crash.

There is work in progress to update the LLVM used by the Darwin stdenv. In the meantime, the easiest fix is to use stdenv.cc on Darwin instead of clang_16. Otherwise, you’ll need to make sure all of your dependencies are linking against libc++ 16 instead of the one in the Darwin stdenv.

https://github.com/NixOS/nixpkgs/issues/234710

2 Likes
6

Thank you. We had got as far as getting otool to show us multiple versions on libc++, but then we didn’t really know what to do with it.

This seems to solve the problem.

How can I keep clang on Linux, without too much code repetition?

I don’t see how something pkgs.stdenv.is{Darwin,Linux} can be used to distinguish between these two versions:

devShell = pkgs.mkShell.override { stdenv = pkgs.clang_16.stdenv; } { ... }
devShell = pkgs.mkShell.override {                                } { ... }

or these two:

devShell = pkgs.mkShell.override { stdenv = pkgs.clang_16.stdenv; } { ... }
devShell = pkgs.mkShell                                             { ... }

without having to repeat the contents of the { ... }.

Even with repetition, is{Darwin,Linux} return empty or non-empty arrays, and devShell needs a single value.

1 Like
7

This works for me to use clang 11 on Darwin and clang 16 on Linux.

diff --git a/flake.nix b/flake.nix
index fb26f92..01c8156 100644
--- a/flake.nix
+++ b/flake.nix
@@ -21,10 +21,12 @@
           enablePython         = false;
           enableRaytracerX11   = false;
         });
-
+        mkShell = pkgs.mkShell.override {
+          stdenv = if pkgs.stdenv.isDarwin then pkgs.stdenv else pkgs.llvmPackages_16.stdenv;
+        };
       in {
 
-        devShell = pkgs.mkShell.override { stdenv = pkgs.clang_16.stdenv; } {
+        devShell = mkShell {
           name = "G4-examples-devenv";
 
           packages = with pkgs; [
@@ -36,7 +38,6 @@
             geant4.data.G4SAIDDATA
             geant4.data.G4PARTICLEXS
             geant4.data.G4NDL
-            clang_16
             clang-tools
             cmake
             cmake-language-server

If you want to use clang 16 on Darwin anyway (before the stdenv update lands), it is possible to use libc++ 11 with clang 16. It’s not pretty though.

diff --git a/flake.nix b/flake.nix
index fb26f92..d1320c9 100644
--- a/flake.nix
+++ b/flake.nix
@@ -22,9 +22,28 @@
           enableRaytracerX11   = false;
         });
 
+        clang_16 = if pkgs.stdenv.isDarwin
+          then pkgs.llvmPackages_16.clang.override rec {
+            libc = pkgs.darwin.apple_sdk.Libsystem;
+            bintools = pkgs.bintools.override { inherit libc; };
+            inherit (pkgs.llvmPackages) libcxx;
+            extraPackages = [
+              pkgs.llvmPackages.libcxxabi
+              # Use the compiler-rt associated with clang, but use the libc++abi from the stdenv
+              # to avoid linking against two different versions (for the same reasons as above).
+              (pkgs.llvmPackages_16.compiler-rt.override {
+                inherit (pkgs.llvmPackages) libcxxabi;
+              })
+            ];
+          }
+          else pkgs.llvmPackages.clang;
+
+        mkShell = pkgs.mkShell.override {
+          stdenv = pkgs.overrideCC pkgs.llvmPackages_16.stdenv clang_16;
+        };
       in {
 
-        devShell = pkgs.mkShell.override { stdenv = pkgs.clang_16.stdenv; } {
+        devShell = mkShell {
           name = "G4-examples-devenv";
 
           packages = with pkgs; [
@@ -36,7 +55,6 @@
             geant4.data.G4SAIDDATA
             geant4.data.G4PARTICLEXS
             geant4.data.G4NDL
-            clang_16
             clang-tools
             cmake
             cmake-language-server
2 Likes
8

Many thanks for your lucid diagnosis and your clear and explicit instructions on how to fix things.

We have wasted many hours on grappling with this whole problem, and you’ve got us past something that had us completely stymied.

Much appreciated.

3 Likes
9

I tried this earlier, but it turns out that other changes resulted in it not being evaluated at all, so I mistakenly believed that it had worked.

It looks like there is a slight error: the apple_sdk in darwin.apple_sdk.Libsystem should not be there.

1 Like
10

I made those changes to the master branch and followed the instructions in the repo to do nix develop and build the example. The clang I got was the requested version, and I was able to build and run the example.

The libc stuff may not be needed (I’ve gotten used to doing it defensively while working on the stdenv), but it shouldn’t cause any errors. Is that an evaluation error or an error while building or at runtime?

1 Like
11

Curious.

I get an evaluation error. Poking around in nix repl shows that pkgs.darwin.apple_sdk.Libsystem does not exist, while pkgs.darwin.Libsystem does. So I removed the apple_sdk and everything worked.

Here is an example of the error appearing on GHA.

2 Likes
12

That is weird. I don’t know why I was able to build it locally. Glad you were able to find a solution.

13

I gladly suffer the injuries to my forehead caused banging my head against Nix, largely to avoid the it-worked-on-my-machine problem, so it always freaks me out a bit when it happens on Nix. And it does. Many orders of magnitude less than anywhere else, but it still does.

Anyway, many thanks again for your help. Very useful.

2 Likes