If a custom kernel build takes that long there’s definitely something wrong. My guess would be that you’re using the default config, which pratically builds every kernel module, including those that are completely unrelated to a raspberry pi.
See this: