Oh I know what happened. substituteInPlace is a shell function so I assume you actually had something like
postPatch = ''
substituteInPlace pom.xml \
--replace "\${gatk.shell.directory}/check_utils_engine_tools.sh" "..."
''
This means you need to worry about both Nix and Bash variable escaping. This can be handled by using your original ''$ escape but adding the \ for bash, as in
postPatch = ''
substituteInPlace pom.xml \
--replace "\''${gatk.shell.directory}/check_utils_engine_tools.sh" "..."
''
Alternatively you could change the " to ' but I’m not sure offhand how to do that given the ''$ immediately following.