macOS Catalina performance issue

Some news broke on HN this week (https://news.ycombinator.com/item?id=23273247, https://news.ycombinator.com/item?id=23281564) about performance issues in macOS Catalina. I spent a little time exploring how one specific issue might affect Nix builds.

I assume this will mature into a GH issue fairly quickly, but first I’m hoping to feel out how much this affects other Catalina users (in particular, it’d be nice to hear from anyone with a multi-user install) and see if there are other thoughts on how to address the issue. (I do think it should be addressed, but I’m not certain how, yet.)

The issue itself

The system is running a blocking networked pre-execution “assessment” of apps and scripts (guessing just unsigned ones?). Notes:

  • The linked HN threads/posts do a decent job of plumbing what is getting sent where, so I won’t address them here.
  • The first check has full TLS overhead, but it keeps the connection open for a minute, so additional checks only add a fraction of the latency per assessment.
  • Naive tests with writing and directly executing a shell script (as laid out in the HN threads) suggests that it caches the result of this check, but it doesn’t appear to be by path or hash–it appears to be doing it by inode. I’m not, however, sure how to square it with the fact that I see multiple checks for the same executables in a Nix build I used to test this. There’s at least one asterisk here I don’t understand yet.

My results

I used the CI build for one of my projects as a test-case. That’s fairly narrow–so I don’t know how well my observations generalize to other builds–I’m curious what others see with similar tests. The assessments appear to make my fresh builds (i.e., post nix-collect-garbage -d) take ~20% longer, but only add ~13% to subsequent builds.

platform assessed+fresh assessed+2nd exempt+fresh exempt+2nd
Early 2020 MBA - Catalina 115.255s 34.777s 97.108s 30.620s
Late 2018 MBA - Catalina 256.54s 50.037s 209.02s 44.196s
Mid 2013 MBA - Mojave n/a n/a 294.679s 66.158s

For additional context, my last few Travis-CI macOS (mac-stadium) builds (iirc on macOS 10.13) took 284s to 295s (not sure how much overhead outside of the execution is lurking here, though).

How to explore it yourself

I think I’ve identified a decent process for exploring this. I recommend using one terminal tab/window to monitor logs, and another to run a build prefixed with the time command (at least once with the assessment enabled, and once without). You can also see the logs in Console.app, but I haven’t refined the queries as much there. The following code block just has 3 different logging commands that do a pretty good job of curating the output to avoid the firehose. You can also swap out stream for show to run a one-time log query, and use the --start and --end flags to bound the search.

# see paths checked
log stream --debug --info --predicate 'process == "syspolicyd" AND subsystem == "com.apple.securityd" AND category == "gk"' 
# see network request summaries
log stream --debug --info --predicate 'process == "syspolicyd" AND subsystem == "com.apple.CFNetwork" AND category == "Summary"'

# see both
log stream --debug --info --predicate 'process == "syspolicyd" AND ((subsystem == "com.apple.securityd" AND category == "gk") OR (subsystem == "com.apple.CFNetwork" AND category == "Summary"))'

If you don’t see any entries here while running your build, it’s possible you’ve already exempted something in the process tree. If so, you can roughly reverse the following instructions for disabling these checks, but you should reboot after doing so. To disable the checks you’ll need to add an exemption for one of the responsible programs to the exemptions at System Preferences > Security & Privacy > Developer Tools (you can hopefully open this by running open -n "x-apple.systempreferences:com.apple.preference.security?Privacy_DevTools"):

  • Single-user install: add your terminal app to the list of exemptions. You shouldn’t need to reboot when you add it–only if you turn the exemption off. (If you don’t have a Developer Tools pane, see point 5 in the final section.)
  • Multi-user install: I haven’t tested this case (I’ll fill it in when someone reports what does/doesn’t work), but I assume you’ll either need to add nix-daemon or launchd. If you add the daemon I assume you’ll have to restart it. If you add launchd, I assume you’ll need to reboot (but, it would be nice to know if it’s not necessary).

How to fix the problem

I’m not sure what the most ergonomic way through this is. It doesn’t look like something Nix can directly “fix”, but I think the performance penalty is big enough that it seems good to make sure every macOS user finds out. Some notes:

  1. I don’t think narrowly exempting the Nix executable is viable. I tried this, but macOS resolves the symlink down to the precise Nix executable, so the exemption wouldn’t be durable.

  2. Exempting the nix-daemon seems more plausible; it’ll be good to hear how this works in practice. My working assumption is that it’ll have the same problem and we’d need to exempt launchd for it to be durable.

  3. If a narrow exemption doesn’t seem viable, there are meta questions like: how appropriate is it to suggest a broader exemption (terminal app, launchd, etc.)? how easy should we make it? etc.

  4. If Nix is going to take any automatic action, it’ll probably need a reliable way to figure out which users need an exemption (and, first, figuring out which executable we’d even be trying to identify an exemption for?). It seems like this will be tricky for single-user installs, but may be straightforward for multi-user ones.

    • These exemptions are stored in an sqlite3 database at /Library/Application Support/com.apple.TCC/TCC.db, so it looks like we can check for existing exemptions with something like: sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "select client,allowed from access where service = 'kTCCServiceDeveloperTool'" (client will be the app ID or the full path of the exempted executable). Here’s an example exemption for Terminal.app:

                              service = kTCCServiceDeveloperTool
                               client = com.apple.Terminal
                          client_type = 0
                              allowed = 1
                         prompt_count = 1
                                csreq = 
                            policy_id = 
      indirect_object_identifier_type = 
           indirect_object_identifier = UNUSED
        indirect_object_code_identity = 
                                flags = 0
                        last_modified = 1590250348
      
    • It looks like there’s a new API, but my logs are showing a message (Service kTCCServiceDeveloperTool does not allow prompting; returning denied.) that makes me think using it to request permission has no effect at the moment. This might, however, be a more stable/sanctioned way to check for an exemption?

    I noticed Nixpkgs already has a package for a python<->objc API named pyobjc (pypi, GitHub) with a corresponding ExecutionPolicy package (doc, pypi sub-package, source), but I had trouble getting this package to build and have no clue what hoops we’d have to jump through to use it to query. There are some usage examples in the tests: 1 2

    • If the others prove unworkable/unstable, it’s also probably possible to “test” for this by using one of the log predicates demonstrated earlier, intentionally run a specific script, and see the path turns up in the next log line.
  5. Regardless of whether it has to be very manual or can be somewhat automatic, I think I’ve sketched out the basics of a tolerable workflow for prompting the user:

    # Terminal was already in my Developer Tools section, but I've seen some
    # post that they don't have a Developer Tools section at all. This command
    # supposedly adds it (though I've seen others assert it doesn't). Note:
    # it doesn't *enable* it. It should just add the section with Terminal 
    # as an unchecked option.
    spctl developer-mode enable-terminal
    
    # Depending on how this runs, we may also need further instructions here.
    # this should open the Security & Privacy settings to the correct view
    # -n forces it to open a new window in case you already have one. 
    # -W will wait for the app to close before continuing
    open -nW "x-apple.systempreferences:com.apple.preference.security?Privacy_DevTools"
    
    # at this point you need to unlock/auth, and then add your exemptions 
    # single-user: toggle your terminal if present, add it with the + button, 
    #              or drag-and-drop it from finder. If there's a good way to figure out 
    #              the path of their terminal, I guess we can open Finder to its folder.
    #  multi-user: use the same process for nix-daemon and/or launchd
    
3 Likes

As far as I understand there are two major factors that affect the impact:

  • Network latency to the Apple server that is used in the check (ocsp.apple.com).
  • Apparently syspolicyd only handles one request at a time (maybe it’s single threaded?). So, requests for verification pile up when there are many in a short amount of time (which is probably more common in builds, e.g. autotools).

Would this approach work when running Nix as a daemon (multi-user)? In single-user, all Nix processes are in the process tree of the terminal application, but I assume that this is not true when the Nix daemon is started through launchd?

Would this approach work when running Nix as a daemon (multi-user)? In single-user, all Nix processes are in the process tree of the terminal application, but I assume that this is not true when the Nix daemon is started through launchd ?

Thanks for calling this out. It would be good to get some reports from people using multi-user installs.

I had this thought at one point while I was working through everything but it slipped my mind when I fell down the rabbit-hole of trying to get pyobjc working. I guess either nix-daemon and/or launchd also need an exemption. I went through my logs and found assessment entries for utils that run during my backup jobs–it’s not quite the same as a build, but I assume nix-daemon users will find the same?

I’ve updated the initial post… :slight_smile:

Exempting launchd seems like a rather bad idea as that will presumably kill this security feature for a large swath of processes or even all of them, depending on precisely what logic the system uses to determine how to apply exemptions (e.g. if it walks up the entire ppid chain then launchd is the ultimate root of every single process).

Indeed. It would obviously be taking a sledgehammer to the problem.

FWIW, I consider even exempting Terminal.app or iterm2.app to be a crude and indiscriminate approach.

To be fair, Apple wasn’t exactly surgical when they shimmed a blocking network request into every execution (presumably by slipping it in a syscall?) without providing any usable API for requesting exemptions or even giving the end user notice of the performance penalty (let alone an actionable notice).

This may be moot.

Hopefully someone else can confirm or refute this result, but I set a spare system up this evening and have yet to find any way to exempt nix-daemon builds. I currently have exemptions set up for all of the below and am still seeing log entries for my builds:

  • com.apple.Terminal
  • /bin/bash
  • /bin/zsh
  • /sbin/launchd
  • /nix/store/*-nix-2.4pre*/bin/nix
  • /nix/store/*-nix-2.4pre*/bin/nix-daemon

Aside: I didn’t run them enough to have a sense of whether the averages were converging, but run-times for assessed daemon builds were close enough to the assessed single-user builds that I suspect they’d converge if I did.

This makes me wonder if the exemptions are only checked for GUI processes. If you exempt /bin/bash and nothing else, do you still get logs when using /bin/bash to run a new script in Terminal.app?

If it is just GUI processes then I expect there’s no GUI process in the chain for nix-daemon.

I gave this a try this morning and it seems like this hunch is right (or more subtle differences between executables determine whether they can be opted out).

I’m not already familiar with the code but I took a few minutes to poke around to see if the source of any of the log messages I identified is public, and did find it in someone’s mirror of Apple’s open source repo.

I haven’t been able to spot or figure out where the exemption is being recognized in order to skip the assessment. The service name in TCC.db for these exemptions is kTCCServiceDeveloperTool, but I don’t see it in this codebase (and there are very few public references to it at all) so I guess there’s relevant code elsewhere deciding whether or not to check at all?

I identified 2 more ways to disable the assessments that do (kinda) work on both single and multi-user installs:

  • Disable assessments before building with sudo spctl --master-disable and re-enable after with sudo spctl --master-enable.
  • Run the build with sudo (directly with sudo nix-build <blah>, in a shell started with sudo, etc.). On a single-user install this works but unfortunately messes up store permissions.

I’ve stalled out a little here as I haven’t had much time to run down more details (in particular, how much this is affecting real workloads, especially since macOS updated from 10.15.4 to 10.15.5 since I did the initial runs here).

If you have a project that does/can use GH actions and does/can build on macOS, it’d be great if you could take a few minutes add a separate job that disables syspolicy assessments before running so that data points can start building up. Here’s how I disabled syspolicy assessments when I moved a project to actions this weekend.

(Unfortunately, I’ve noticed pretty big swings in how long my macOS jobs take to complete. It may take a fair number of data points and some attempt at normalizing them…)

I’ve finally gotten around to opening an issue about this at https://github.com/NixOS/nix/issues/3789.