I have no experience about what you are trying to achieve, but rdma and related technologies (infiniband, qlogic, sr-iov, ROCE) is not it. These are network technologies that permit high bandwidth/low latency data transfer between hosts. Most of these bypass the IP stack entirely.
Infiniband is a network stack that enable RDMA, it’s only vendor is now NVIDIA which acquired mellanox. Qlogic was another vendor, but it got acquired by Intel that tried to market it as Omnipath, but it was spinned off to Cornelis network.
Sr-iov is a way to share an infiniband card to a virtual machine on the same host.
ROCE is an implementation of the rdma software stack over ethernet instead of infiniband.
If both networks 10.100.100.0/24? And 10.20.20.0/24 share the same level 2 Ethernet segment/vlan/broadcast domain, you don’t even need the third nic, you can setup a secondary IPv4 address on the private nic on the 10.20.20.0/24 network.
I would not call that best practice, but if the number of host on the network is reasonable and you are aware of the security problems created, there’s nothing really wrong with this setup.
Having two nics on the same Ethernet network is actually trickier since you have to do ARP filtering.