Basic Infiniband setup for those interested:
hardware.infiniband.enable = true;
hardware.infiniband.guids = ["0x0002c90300a39cc1" "0x0002c90300a39cc2"];
This is kinda sorta documented Appendix A. Configuration Options
What is missing is that the “guids” (Infiniband ports" are required otherwise the subnet manger (OpenSM) will not be setup. The nix file for setting up hardware.infiniband also sets up the subnet manager and requires the ib guids as part of the command line in the nix configuration.
You can get the ib guids from ibstat or ibstat -p to just get the guids.
After you run nixos-rebuild (boot or switch) you will have a working subnet manager and then you’ll get link from the IB interface. IPoIB is also ready to be configured if desired.
Setup the new IB interfaces pretty much the same as you would any other network interface. IPoIB requires a IP gateway capable switch. If you don’t plan on using IPoIB then continue on like any other Linux system configuring your RDMA capable apps or filesystem.