How to add a native dependency to gem builds

I’ve got a gem which includes a native extension and requires linking with a library. Unfortunately I can’t find a way to get it to find / link with the library.
There doesn’t seem to be a clear way to add that library and pkg-config as build dependencies in bundlerEnv. How do people deal with that situation?