Hm, interesting! It turns out setxkbmap demands a full directory structure as input (and there’s no separate flag for symbols, unlike rules). You could create a fake directory structure, but that’d be besides the point of a minimal test of what NixOS actually does.
So let’s look at the implementation.
The config is created in a custom package here, through a rather horrible-looking wrapper derivation.
While the derivation there is fairly complex, it looks like it should set up your configuration correctly, so looking at how this is used we can see an environment variable set by the module a bit further down, which points at the directory that NixOS creates for you - this means we can actually test this rather conveniently like so:
setxkbmap -I"${XKB_CONFIG_ROOT}" us-custom -print | xkbcomp -I"${XKB_CONFIG_ROOT}" - $DISPLAY
Sleuthing through that path and trying out the manual commands should help finding whatever is fishy here, and is probably the quickest way to figure out why this doesn’t work. It’ll likely be something silly like a mis-named file.
If not, it seems that the way this is applied is through the services.xserver.xkbDir setting, which is apparently passed directly to the X binary as a CLI argument - since you’re launching X with a display manager, it in turn will not be managed by systemd, and we indeed don’t expect any useful output from the display manager unit (except listing the arguments of the spawned sub-processes, which is another way of figuring out where the config ended up).
Instead, any errors with this will be reported by Xorg itself - Xorg will create a bunch of log files in /var/log/X.*.log, where there may be more info.
The fishy thing is basically this here, which will cp a store path during the system build. If I recall correctly, that will generally fail on something like ofborg, or anything a bit stricter, but this should work at least locally, so your problem is probably still something else entirely.
The best practice here would probably be to fix the upstream module, and allow passing raw text as well as file paths, which the derivation could then turn into files by piping them into the build tree. It may well be worth writing a patch for that and opening a PR.