Some news broke on HN this week (MacOS Catalina: Slow by Design? | Hacker News, Catalina is checking notarization of unsigned executables | Hacker News) about performance issues in macOS Catalina. I spent a little time exploring how one specific issue might affect Nix builds.
I assume this will mature into a GH issue fairly quickly, but first I’m hoping to feel out how much this affects other Catalina users (in particular, it’d be nice to hear from anyone with a multi-user install) and see if there are other thoughts on how to address the issue. (I do think it should be addressed, but I’m not certain how, yet.)
The issue itself
The system is running a blocking networked pre-execution “assessment” of apps and scripts (guessing just unsigned ones?). Notes:
- The linked HN threads/posts do a decent job of plumbing what is getting sent where, so I won’t address them here.
- The first check has full TLS overhead, but it keeps the connection open for a minute, so additional checks only add a fraction of the latency per assessment.
- Naive tests with writing and directly executing a shell script (as laid out in the HN threads) suggests that it caches the result of this check, but it doesn’t appear to be by path or hash–it appears to be doing it by inode. I’m not, however, sure how to square it with the fact that I see multiple checks for the same executables in a Nix build I used to test this. There’s at least one asterisk here I don’t understand yet.
My results
I used the CI build for one of my projects as a test-case. That’s fairly narrow–so I don’t know how well my observations generalize to other builds–I’m curious what others see with similar tests. The assessments appear to make my fresh builds (i.e., post nix-collect-garbage -d) take ~20% longer, but only add ~13% to subsequent builds.
| platform | assessed+fresh | assessed+2nd | exempt+fresh | exempt+2nd |
|---|---|---|---|---|
| Early 2020 MBA - Catalina | 115.255s | 34.777s | 97.108s | 30.620s |
| Late 2018 MBA - Catalina | 256.54s | 50.037s | 209.02s | 44.196s |
| Mid 2013 MBA - Mojave | n/a | n/a | 294.679s | 66.158s |
For additional context, my last few Travis-CI macOS (mac-stadium) builds (iirc on macOS 10.13) took 284s to 295s (not sure how much overhead outside of the execution is lurking here, though).
How to explore it yourself
I think I’ve identified a decent process for exploring this. I recommend using one terminal tab/window to monitor logs, and another to run a build prefixed with the time command (at least once with the assessment enabled, and once without). You can also see the logs in Console.app, but I haven’t refined the queries as much there. The following code block just has 3 different logging commands that do a pretty good job of curating the output to avoid the firehose. You can also swap out stream for show to run a one-time log query, and use the --start and --end flags to bound the search.
# see paths checked
log stream --debug --info --predicate 'process == "syspolicyd" AND subsystem == "com.apple.securityd" AND category == "gk"'
# see network request summaries
log stream --debug --info --predicate 'process == "syspolicyd" AND subsystem == "com.apple.CFNetwork" AND category == "Summary"'
# see both
log stream --debug --info --predicate 'process == "syspolicyd" AND ((subsystem == "com.apple.securityd" AND category == "gk") OR (subsystem == "com.apple.CFNetwork" AND category == "Summary"))'
If you don’t see any entries here while running your build, it’s possible you’ve already exempted something in the process tree. If so, you can roughly reverse the following instructions for disabling these checks, but you should reboot after doing so. To disable the checks you’ll need to add an exemption for one of the responsible programs to the exemptions at System Preferences > Security & Privacy > Developer Tools (you can hopefully open this by running open -n "x-apple.systempreferences:com.apple.preference.security?Privacy_DevTools"):
- Single-user install: add your terminal app to the list of exemptions. You shouldn’t need to reboot when you add it–only if you turn the exemption off. (If you don’t have a Developer Tools pane, see point 5 in the final section.)
- Multi-user install: I haven’t tested this case (I’ll fill it in when someone reports what does/doesn’t work), but I assume you’ll either need to add nix-daemon or launchd. If you add the daemon I assume you’ll have to restart it. If you add launchd, I assume you’ll need to reboot (but, it would be nice to know if it’s not necessary).
How to fix the problem
I’m not sure what the most ergonomic way through this is. It doesn’t look like something Nix can directly “fix”, but I think the performance penalty is big enough that it seems good to make sure every macOS user finds out. Some notes:
-
I don’t think narrowly exempting the Nix executable is viable. I tried this, but macOS resolves the symlink down to the precise Nix executable, so the exemption wouldn’t be durable.
-
Exempting the nix-daemon seems more plausible; it’ll be good to hear how this works in practice. My working assumption is that it’ll have the same problem and we’d need to exempt launchd for it to be durable.
-
If a narrow exemption doesn’t seem viable, there are meta questions like: how appropriate is it to suggest a broader exemption (terminal app, launchd, etc.)? how easy should we make it? etc.
-
If Nix is going to take any automatic action, it’ll probably need a reliable way to figure out which users need an exemption (and, first, figuring out which executable we’d even be trying to identify an exemption for?). It seems like this will be tricky for single-user installs, but may be straightforward for multi-user ones.
-
These exemptions are stored in an sqlite3 database at /Library/Application Support/com.apple.TCC/TCC.db, so it looks like we can check for existing exemptions with something like:
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "select client,allowed from access where service = 'kTCCServiceDeveloperTool'"(client will be the app ID or the full path of the exempted executable). Here’s an example exemption for Terminal.app:service = kTCCServiceDeveloperTool client = com.apple.Terminal client_type = 0 allowed = 1 prompt_count = 1 csreq = policy_id = indirect_object_identifier_type = indirect_object_identifier = UNUSED indirect_object_code_identity = flags = 0 last_modified = 1590250348 -
It looks like there’s a new API, but my logs are showing a message (
Service kTCCServiceDeveloperTool does not allow prompting; returning denied.) that makes me think using it to request permission has no effect at the moment. This might, however, be a more stable/sanctioned way to check for an exemption?
I noticed Nixpkgs already has a package for a python<->objc API named pyobjc (pypi, GitHub) with a corresponding ExecutionPolicy package (doc, pypi sub-package, source), but I had trouble getting this package to build and have no clue what hoops we’d have to jump through to use it to query. There are some usage examples in the tests: 1 2
- If the others prove unworkable/unstable, it’s also probably possible to “test” for this by using one of the log predicates demonstrated earlier, intentionally run a specific script, and see the path turns up in the next log line.
-
-
Regardless of whether it has to be very manual or can be somewhat automatic, I think I’ve sketched out the basics of a tolerable workflow for prompting the user:
# Terminal was already in my Developer Tools section, but I've seen some # post that they don't have a Developer Tools section at all. This command # supposedly adds it (though I've seen others assert it doesn't). Note: # it doesn't *enable* it. It should just add the section with Terminal # as an unchecked option. spctl developer-mode enable-terminal # Depending on how this runs, we may also need further instructions here. # this should open the Security & Privacy settings to the correct view # -n forces it to open a new window in case you already have one. # -W will wait for the app to close before continuing open -nW "x-apple.systempreferences:com.apple.preference.security?Privacy_DevTools" # at this point you need to unlock/auth, and then add your exemptions # single-user: toggle your terminal if present, add it with the + button, # or drag-and-drop it from finder. If there's a good way to figure out # the path of their terminal, I guess we can open Finder to its folder. # multi-user: use the same process for nix-daemon and/or launchd
